diff options
Diffstat (limited to 'doc')
691 files changed, 15477 insertions, 8090 deletions
diff --git a/doc/.vale/gitlab/Uppercase.yml b/doc/.vale/gitlab/Uppercase.yml index bdbee8108d5..a8519d898db 100644 --- a/doc/.vale/gitlab/Uppercase.yml +++ b/doc/.vale/gitlab/Uppercase.yml @@ -16,6 +16,7 @@ second: '(?:\b[A-Z][a-z]+ )+\(([A-Z]{3,5})\)' exceptions: - ACL - AJAX + - AMI - ANSI - APAC - API @@ -23,6 +24,7 @@ exceptions: - ARM - ARN - ASCII + - ASG - AWS - BMP - BSD @@ -58,11 +60,13 @@ exceptions: - DSL - DVCS - DVD + - EBS - ECDSA - ECS - EFS - EKS - ELB + - ENA - EOL - EWM - EXIF @@ -88,6 +92,7 @@ exceptions: - GPG - GPL - GPS + - GPT - GPU - GUI - HAML @@ -125,10 +130,12 @@ exceptions: - LSIF - LTM - LTS + - LVM - MIME - MIT - MITRE - MVC + - NAS - NAT - NDA - NFS diff --git a/doc/.vale/gitlab/spelling-exceptions.txt b/doc/.vale/gitlab/spelling-exceptions.txt index de7fb45490d..43ccaf6bd7a 100644 --- a/doc/.vale/gitlab/spelling-exceptions.txt +++ b/doc/.vale/gitlab/spelling-exceptions.txt @@ -125,6 +125,7 @@ colocated colocating compilable composable +composables Conda Consul Contentful @@ -239,6 +240,7 @@ Fortinet Fugit fuzzer Gantt +Gbps Gemfile Gemnasium Gemojione diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md index ad235ead992..817f22debbc 100644 --- a/doc/administration/audit_event_streaming.md +++ b/doc/administration/audit_event_streaming.md @@ -9,6 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. > - [Enabled on GitLab.com and by default on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338939) in GitLab 14.7. > - [Feature flag `ff_external_audit_events_namespace`](https://gitlab.com/gitlab-org/gitlab/-/issues/349588) removed in GitLab 14.8. +> - [Subgroup events recording](https://gitlab.com/gitlab-org/gitlab/-/issues/366878) fixed in GitLab 15.2. Users can set an HTTP endpoint for a top-level group to receive all audit events about the group, its subgroups, and projects as structured JSON. @@ -29,11 +30,11 @@ Event streaming destinations receive **all** audit event data, which could inclu Users with at least the Owner role for a group can add event streaming destinations for it: 1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events** +1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. - - When the destination list is empty, select **Add stream** activate edit mode and add a new destination. - - When the destination list is not empty, select **{plus}** under the **Streams** tab to activate edit mode. -1. Enter the endpoint you wish to add and select **Add**. + - When the destination list is empty, select **Add stream** to show the section for adding destinations. + - When the destination list is not empty, select **{plus}** to show the section for adding destinations. +1. Enter the destination URL to add and select **Add**. Event streaming is enabled if: @@ -76,7 +77,7 @@ Users with at least the Owner role for a group can list event streaming destinat Users with at least the Owner role for a group can list event streaming destinations: 1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events** +1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. ### Use the API @@ -115,7 +116,7 @@ When the last destination is successfully deleted, event streaming is disabled f Users with at least the Owner role for a group can delete event streaming destinations. 1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Security & Compliance > Audit events** +1. On the left sidebar, select **Security & Compliance > Audit events**. 1. On the main area, select **Streams** tab. 1. Select **{remove}** at the right side of each item. @@ -143,20 +144,26 @@ Destination is deleted if: - The returned `errors` object is empty. - The API responds with `200 OK`. -## Custom HTTP header values +## Custom HTTP headers -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. +> - API [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361216) in GitLab 15.1 [with a flag](feature_flags.md) named `streaming_audit_event_headers`. Disabled by default. +> - API [enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/362941) in GitLab 15.2. +> - UI [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361630) in GitLab 15.2 [with a flag](feature_flags.md) named `custom_headers_streaming_audit_events_ui`. Disabled by default. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `streaming_audit_event_headers`. -On GitLab.com, this feature is not available. -The feature is not ready for production use. +On self-managed GitLab, by default the API for this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `streaming_audit_event_headers`. +On GitLab.com, the API for this feature is available. Each streaming destination can have up to 20 custom HTTP headers included with each streamed event. -### Add with the API +### Adding custom HTTP headers -Group owners can add a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. +Add custom HTTP headers with the API or GitLab UI. + +#### Use the API + +Group owners can add a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. You can retrieve the destination ID +by [listing the external audit destinations](#list-streaming-destinations) on the group. ```graphql mutation { @@ -166,19 +173,111 @@ mutation { } ``` -### Delete with the API +The header is created if the returned `errors` object is empty. + +#### Use the GitLab UI + +FLAG: +On self-managed GitLab, by default the UI for this feature is not available. To make it available per group, ask an administrator to +[enable the feature flag](../administration/feature_flags.md) named `custom_headers_streaming_audit_events_ui`. On GitLab.com, the UI for this feature is +not available. The UI for this feature is not ready for production use. + +Users with at least the Owner role for a group can add event streaming destinations and custom HTTP headers for it: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the main area, select **Streams** tab. + - When the destination list is empty, select **Add stream** to show the section for adding destinations. + - When the destination list is not empty, select **{plus}** to show the section for adding destinations. +1. Enter the destination URL to add. +1. Locate the **Custom HTTP headers** table. +1. In the **Header** column, add the header's name. +1. In the **Value** column, add the header's value. +1. Ignore the **Active** checkbox because it isn't functional. To track progress on adding functionality to the **Active** checkbox, see the + [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). +1. Enter as many name and value pairs as required. When you enter a unique name and a value for a header, a new row in the table automatically appears. You can add up to + 20 headers per endpoint. +1. After all headers have been filled out, select **Add** to add the new endpoint. + +Event streaming is enabled if: + +- No warning is shown. +- The added endpoint is displayed in the UI. + +### Updating custom HTTP headers + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/361964) in GitLab 15.2. -Group owners can remove a HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. +Group owners can update a HTTP header using the GraphQL `auditEventsStreamingHeadersCreate` mutation. ```graphql mutation { - auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::ExternalAuditEventDestination/24601" }) { + auditEventsStreamingHeadersUpdate(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/24255", key: "new-foo", value: "new-bar" }) { errors } } ``` -The header is created if the returned `errors` object is empty. +### Deleting custom HTTP headers + +Group owners can remove a HTTP header using the GraphQL `auditEventsStreamingHeadersDestroy` mutation. You can retrieve the header ID +by [listing all the custom headers](#list-all-custom-headers) on the group. + +```graphql +mutation { + auditEventsStreamingHeadersDestroy(input: { headerId: "gid://gitlab/AuditEvents::Streaming::Header/1" }) { + errors + } +} +``` + +The header is deleted if the returned `errors` object is empty. + +### List all custom headers + +List all custom HTTP headers with the API or GitLab UI. + +#### Use the API + +You can list all custom headers for a top-level group as well as their value and ID using the GraphQL `externalAuditEventDestinations` query. The ID +value returned by this query is what you need to pass to the `deletion` mutation. + +```graphql +query { + group(fullPath: "your-group") { + id + externalAuditEventDestinations { + nodes { + destinationUrl + id + headers { + nodes { + key + value + id + } + } + } + } + } +} +``` + +#### Use the GitLab UI + +FLAG: +On self-managed GitLab, by default the UI for this feature is not available. To make it available per group, ask an administrator to +[enable the feature flag](../administration/feature_flags.md) named `custom_headers_streaming_audit_events_ui`. On GitLab.com, the UI for this feature is +not available. The UI for this feature is not ready for production use. + +Users with at least the Owner role for a group can add event streaming destinations and custom HTTP headers for it: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the main area, select **Streams** tab. +1. Select **{pencil}** at the right side of an item. +1. A read-only view of the items custom headers is shown. To track progress on adding editing functionality, see the [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/361925). +1. Select **Cancel** to close the read-only view. ## Verify event authenticity @@ -190,6 +289,17 @@ token is generated when the event destination is created and cannot be changed. Each streamed event contains a random alphanumeric identifier for the `X-Gitlab-Event-Streaming-Token` HTTP header that can be verified against the destination's value when [listing streaming destinations](#list-streaming-destinations). +### Use the GitLab UI + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360814) in GitLab 15.2. + +Users with at least the Owner role for a group can list event streaming destinations and see the verification tokens: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Security & Compliance > Audit events**. +1. On the main area, select **Streams**. +1. View the verification token on the right side of each item. + ## Audit event streaming on Git operations > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `audit_event_streaming_git_operations`. Disabled by default. @@ -427,3 +537,215 @@ X-Gitlab-Audit-Event-Type: audit_operation "event_type": "audit_operation" } ``` + +## Audit event streaming on merge request create actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90911) in GitLab 15.2. + +Stream audit events that relate to merge request create actions using the `/logs` endpoint. + +Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value `merge_request_create`. GitLab responds with JSON payloads with an +`event_type` field set to `merge_request_create`. + +### Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Audit-Event-Type: merge_request_create +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example payload + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example_user", + "target_id": 132, + "target_type": "MergeRequest", + "target_details": "Update test.md", + "custom_message": "Added merge request", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "Administrator", + "entity_path": "example-group/example-project", + "target_details": "Update test.md", + "created_at": "2022-07-04T00:19:22.675Z", + "target_type": "MergeRequest", + "target_id": 132, + "event_type": "merge_request_create" +} +``` + +## Audit event streaming on project fork actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90916) in GitLab 15.2. + +Stream audit events that relate to project fork actions using the `/logs` endpoint. + +Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value `project_fork_operation`. GitLab responds with JSON payloads with an +`event_type` field set to `project_fork_operation`. + +### Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Audit-Event-Type: project_fork_operation +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example payload + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example_username", + "target_id": 24, + "target_type": "Project", + "target_details": "example-project", + "custom_message": "Forked project to another-group/example-project-forked", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example_username", + "entity_path": "example-group/example-project", + "target_details": "example-project", + "created_at": "2022-06-30T03:43:35.384Z", + "target_type": "Project", + "target_id": 24, + "event_type": "project_fork_operation" +} +``` + +## Audit event streaming on project group link actions + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90955) in GitLab 15.2. + +Stream audit events that relate to project group link creation, updates, and deletion using the `/logs` endpoint. + +Send API requests that contain the `X-Gitlab-Audit-Event-Type` header with value of either: + +- `project_group_link_create`. +- `project_group_link_update`. +- `project_group_link_destroy`. + +GitLab responds with JSON payloads with an `event_type` field set to either: + +- `project_group_link_create`. +- `project_group_link_update`. +- `project_group_link_destroy`. + +### Example Headers + +Headers are formatted as follows: + +```plaintext +POST /logs HTTP/1.1 +Host: <DESTINATION_HOST> +Content-Type: application/x-www-form-urlencoded +X-Gitlab-Audit-Event-Type: project_group_link_create +X-Gitlab-Event-Streaming-Token: <DESTINATION_TOKEN> +``` + +### Example payload for project group link create + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example-user", + "target_id": 31, + "target_type": "Group", + "target_details": "another-group", + "custom_message": "Added project group link", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example-user", + "entity_path": "example-group/example-project", + "target_details": "another-group", + "created_at": "2022-07-04T00:43:09.318Z", + "target_type": "Group", + "target_id": 31, + "event_type": "project_group_link_create" +} +``` + +### Example payload for project group link update + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example-user", + "target_id": 31, + "target_type": "Group", + "target_details": "another-group", + "custom_message": "Changed project group link profile group_access from Developer to Guest", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example-user", + "entity_path": "example-group/example-project", + "target_details": "another-group", + "created_at": "2022-07-04T00:43:28.328Z", + "target_type": "Group", + "target_id": 31, + "event_type": "project_group_link_update" +} +``` + +### Example payload for project group link delete + +```json +{ + "id": 1, + "author_id": 1, + "entity_id": 24, + "entity_type": "Project", + "details": { + "author_name": "example-user", + "target_id": 31, + "target_type": "Group", + "target_details": "another-group", + "custom_message": "Removed project group link", + "ip_address": "127.0.0.1", + "entity_path": "example-group/example-project" + }, + "ip_address": "127.0.0.1", + "author_name": "example-user", + "entity_path": "example-group/example-project", + "target_details": "another-group", + "created_at": "2022-07-04T00:42:56.279Z", + "target_type": "Group", + "target_id": 31, + "event_type": "project_group_link_destroy" +} +``` diff --git a/doc/administration/auth/index.md b/doc/administration/auth/index.md index ad8d78fdfd6..a412875501e 100644 --- a/doc/administration/auth/index.md +++ b/doc/administration/auth/index.md @@ -25,13 +25,15 @@ UltraAuth has removed their software which supports OmniAuth integration. We hav The external authentication and authorization providers may support the following capabilities. For more information, see the links shown on this page for each external provider. -| Capability | SaaS | Self-Managed | +| Capability | SaaS | Self-managed | |-------------------------------------------------|-----------------------------------------|------------------------------------| -| **User Provisioning** | SCIM<br>Just-In-Time (JIT) Provisioning | LDAP Sync | +| **User Provisioning** | SCIM<br>SAML <sup>1</sup> | LDAP <sup>1</sup><br>SAML <sup>1</sup><br>[OmniAuth Providers](../../integration/omniauth.md#supported-providers) <sup>1</sup> | | **User Detail Updating** (not group management) | Not Available | LDAP Sync | -| **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>OmniAuth Providers (only 1 permitted per unique provider) | +| **Authentication** | SAML at top-level group (1 provider) | LDAP (multiple providers)<br>Generic OAuth2<br>SAML (only 1 permitted per unique provider)<br>Kerberos<br>JWT<br>Smartcard<br>[OmniAuth Providers](../../integration/omniauth.md#supported-providers) (only 1 permitted per unique provider) | | **Provider-to-GitLab Role Sync** | SAML Group Sync | LDAP Group Sync<br>SAML Group Sync ([GitLab 15.1](https://gitlab.com/gitlab-org/gitlab/-/issues/285150) and later) | -| **User Removal** | SCIM (remove user from top-level group) | LDAP (Blocking User from Instance) | +| **User Removal** | SCIM (remove user from top-level group) | LDAP (remove user from groups and block from the instance) | + +1. Using Just-In-Time (JIT) provisioning, user accounts are created when the user first signs in. ## Change apps or configuration diff --git a/doc/administration/auth/ldap/img/multi_login.gif b/doc/administration/auth/ldap/img/multi_login.gif Binary files differdeleted file mode 100644 index 5aee6090793..00000000000 --- a/doc/administration/auth/ldap/img/multi_login.gif +++ /dev/null diff --git a/doc/administration/auth/ldap/img/multi_login.png b/doc/administration/auth/ldap/img/multi_login.png Binary files differnew file mode 100644 index 00000000000..512f403a442 --- /dev/null +++ b/doc/administration/auth/ldap/img/multi_login.png diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 55b57193bf3..05eee338e64 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Integrate LDAP with GitLab **(FREE SELF)** -GitLab integrates with [LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) +GitLab integrates with [LDAP - Lightweight Directory Access Protocol](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol) to support user authentication. This integration works with most LDAP-compliant directory servers, including: @@ -274,7 +274,7 @@ gitlab_rails['ldap_servers'] = { This example results in the following sign-in page: -![Multiple LDAP servers sign in](img/multi_login.gif) +![Multiple LDAP servers sign in](img/multi_login.png) ### Set up LDAP user filter diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md index 0f0d301bfa9..b0ada1c11dd 100644 --- a/doc/administration/auth/ldap/ldap_synchronization.md +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -42,11 +42,6 @@ The process also updates the following user information: - SSH public keys (if `sync_ssh_keys` is set) - Kerberos identity (if Kerberos is enabled) -The LDAP sync process: - -- Updates existing users. -- Creates new users on first sign in. - ### Adjust LDAP user sync schedule By default, GitLab runs a worker once per day at 01:30 a.m. server time to diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index deb47160d98..60a4cc8706f 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -339,7 +339,7 @@ The trailing forward slash is required. This configuration corresponds with the `Supported account types` setting used when creating the `IdentityExperienceFramework` app. -#### Keycloak +### Keycloak GitLab works with OpenID providers that use HTTPS. Although a Keycloak server can be set up using HTTP, GitLab can only communicate @@ -380,7 +380,7 @@ gitlab_rails['omniauth_providers'] = [ ] ``` -##### Configure Keycloak with a symmetric key algorithm +#### Configure Keycloak with a symmetric key algorithm > Introduced in GitLab 14.2. @@ -461,7 +461,7 @@ To use symmetric key encryption: If after reconfiguring, you see the error `JSON::JWS::VerificationFailed` error message, this means the incorrect secret was specified. -#### Casdoor +### Casdoor GitLab works with OpenID providers that use HTTPS. To connect to GitLab using OpenID with Casdoor, use HTTPS instead of HTTP. @@ -471,7 +471,7 @@ For your app, complete the following steps on Casdoor: 1. Add your GitLab redirect URL. For example, if your GitLab domain is `gitlab.example.com`, ensure the Casdoor app has the following `Redirect URI`: `https://gitlab.example.com/users/auth/openid_connect/callback`. -See the [Casdoor documentation](https://casdoor.org/docs/integration/gitlab/) for more details. +See the [Casdoor documentation](https://casdoor.org/docs/integration/gitlab) for more details. Example Omnibus GitLab configuration (file path: `/etc/gitlab/gitlab.rb`): diff --git a/doc/administration/consul.md b/doc/administration/consul.md index e1f2bd90e05..e282960c857 100644 --- a/doc/administration/consul.md +++ b/doc/administration/consul.md @@ -80,14 +80,14 @@ Nodes should be: - Upgraded one node at a time. Identify any existing health issues in the cluster by running the following command -within each node. The command will return an empty array if the cluster is healthy: +in each node. The command returns an empty array if the cluster is healthy: ```shell curl "http://127.0.0.1:8500/v1/health/state/critical" ``` -If the Consul version has changed, you'll see a notice at the end of `gitlab-ctl reconfigure` -informing you that Consul needs to be restarted for the new version to be used. +If the Consul version has changed, you see a notice at the end of `gitlab-ctl reconfigure` +informing you that Consul must be restarted for the new version to be used. Restart Consul one node at a time: @@ -96,7 +96,7 @@ sudo gitlab-ctl restart consul ``` Consul nodes communicate using the raft protocol. If the current leader goes -offline, there needs to be a leader election. A leader node must exist to facilitate +offline, there must be a leader election. A leader node must exist to facilitate synchronization across the cluster. If too many nodes go offline at the same time, the cluster loses quorum and doesn't elect a leader due to [broken consensus](https://www.consul.io/docs/architecture/consensus). @@ -111,7 +111,7 @@ bundled Consul wasn't used by any process other than GitLab itself, you can ## Troubleshooting Consul -Below are some useful operations should you need to debug any issues. +Below are some operations should you debug any issues. You can see any error logs by running: ```shell @@ -149,7 +149,7 @@ To be safe, it's recommended that you only restart Consul in one node at a time ensure the cluster remains intact. For larger clusters, it is possible to restart multiple nodes at a time. See the [Consul consensus document](https://www.consul.io/docs/architecture/consensus#deployment-table) -for the number of failures it can tolerate. This will be the number of simultaneous +for the number of failures it can tolerate. This is the number of simultaneous restarts it can sustain. To restart Consul: diff --git a/doc/administration/docs_self_host.md b/doc/administration/docs_self_host.md index 338601a054f..3f5d4adc95c 100644 --- a/doc/administration/docs_self_host.md +++ b/doc/administration/docs_self_host.md @@ -149,7 +149,7 @@ To extract the HTML files of the Docs site: docker cp gitlab-docs:/usr/share/nginx/html /srv/gitlab/ ``` - You will end up with a `/srv/gitlab/html/` directory that holds the documentation website. + You end up with a `/srv/gitlab/html/` directory that holds the documentation website. 1. Remove the container: diff --git a/doc/administration/encrypted_configuration.md b/doc/administration/encrypted_configuration.md index 38f033e260a..a96a4c4405d 100644 --- a/doc/administration/encrypted_configuration.md +++ b/doc/administration/encrypted_configuration.md @@ -14,12 +14,12 @@ GitLab can read settings for certain features from encrypted settings files. The - [LDAP `bind_dn` and `password`](auth/ldap/index.md#use-encrypted-credentials). - [SMTP `user_name` and `password`](raketasks/smtp.md#secrets). -In order to enable the encrypted configuration settings, a new base key needs to be generated for +To enable the encrypted configuration settings, a new base key must be generated for `encrypted_settings_key_base`. The secret can be generated in the following ways: **Omnibus Installation** -Starting with 13.7 the new secret is automatically generated for you, but you need to ensure your +Starting with 13.7 the new secret is automatically generated for you, but you must ensure your `/etc/gitlab/gitlab-secrets.json` contains the same values on all nodes. **GitLab Cloud Native Helm Chart** diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index 8a021c6d588..77f7f621a07 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -64,7 +64,7 @@ the status of the flag and the command to enable or disable it. ### Start the GitLab Rails console -The first thing you need to enable or disable a feature behind a flag is to +The first thing you must do to enable or disable a feature behind a flag is to start a session on GitLab Rails console. For Omnibus installations: @@ -83,7 +83,7 @@ For details, see [starting a Rails console session](operations/rails_console.md# ### Enable or disable the feature -Once the Rails console session has started, run the `Feature.enable` or +After the Rails console session has started, run the `Feature.enable` or `Feature.disable` commands accordingly. The specific flag can be found in the feature's documentation itself. @@ -123,11 +123,11 @@ For example, to enable the [`:product_analytics`](../operations/product_analytic Feature.enable(:product_analytics, Project.find(1234)) ``` -`Feature.enable` and `Feature.disable` always return `nil`, this is not an indication that the command failed: +`Feature.enable` and `Feature.disable` always return `true`, even if the application doesn't use the flag: ```ruby irb(main):001:0> Feature.enable(:my_awesome_feature) -=> nil +=> true ``` When the feature is ready, GitLab removes the feature flag, and the option for @@ -159,7 +159,7 @@ Feature.all ### Unset feature flag -You can unset a feature flag so that GitLab will fall back to the current defaults for that flag: +You can unset a feature flag so that GitLab falls back to the current defaults for that flag: ```ruby Feature.remove(:my_awesome_feature) diff --git a/doc/administration/geo/disaster_recovery/background_verification.md b/doc/administration/geo/disaster_recovery/background_verification.md index 57f8664ba11..97c9a6c5576 100644 --- a/doc/administration/geo/disaster_recovery/background_verification.md +++ b/doc/administration/geo/disaster_recovery/background_verification.md @@ -13,29 +13,25 @@ disable or enable this feature manually by following [these instructions](#disabling-or-enabling-the-automatic-background-verification). Automatic background verification ensures that the transferred data matches a -calculated checksum. If the checksum of the data on the **primary** node matches checksum of the -data on the **secondary** node, the data transferred successfully. Following a planned failover, +calculated checksum. If the checksum of the data on the **primary** site matches checksum of the +data on the **secondary** site, the data transferred successfully. Following a planned failover, any corrupted data may be **lost**, depending on the extent of the corruption. -If verification fails on the **primary** node, this indicates Geo is replicating a corrupted object. -You can restore it from backup or remove it from the **primary** node to resolve the issue. +If verification fails on the **primary** site, this indicates Geo is replicating a corrupted object. +You can restore it from backup or remove it from the **primary** site to resolve the issue. -If verification succeeds on the **primary** node but fails on the **secondary** node, +If verification succeeds on the **primary** site but fails on the **secondary** site, this indicates that the object was corrupted during the replication process. Geo actively try to correct verification failures marking the repository to be resynced with a back-off period. If you want to reset the verification for these failures, so you should follow [these instructions](background_verification.md#reset-verification-for-projects-where-verification-has-failed). If verification is lagging significantly behind replication, consider giving -the node more time before scheduling a planned failover. +the site more time before scheduling a planned failover. ## Disabling or enabling the automatic background verification -Run the following commands in a Rails console on the **primary** node: - -```shell -gitlab-rails console -``` +Run the following commands in a [Rails console](../../operations/rails_console.md) on a **Rails node on the primary** site. To check if automatic background verification is enabled: @@ -57,33 +53,33 @@ Feature.enable('geo_repository_verification') ## Repository verification -On the **primary** node: +On the **primary** site: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Expand **Verification information** tab for that node to view automatic checksumming +1. On the left sidebar, select **Geo > Sites**. +1. Expand **Verification information** tab for that site to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work in gray, and failures in red. ![Verification status](img/verification_status_primary_v14_0.png) -On the **secondary** node: +On the **secondary** site: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Expand **Verification information** tab for that node to view automatic checksumming +1. On the left sidebar, select **Geo > Sites**. +1. Expand **Verification information** tab for that site to view automatic checksumming status for repositories and wikis. Successes are shown in green, pending work in gray, and failures in red. ![Verification status](img/verification_status_secondary_v14_0.png) -## Using checksums to compare Geo nodes +## Using checksums to compare Geo sites -To check the health of Geo **secondary** nodes, we use a checksum over the list of +To check the health of Geo **secondary** sites, we use a checksum over the list of Git references and their values. The checksum includes `HEAD`, `heads`, `tags`, -`notes`, and GitLab-specific references to ensure true consistency. If two nodes +`notes`, and GitLab-specific references to ensure true consistency. If two sites have the same checksum, then they definitely hold the same references. We compute -the checksum for every node after every update to make sure that they are all +the checksum for every site after every update to make sure that they are all in sync. ## Repository re-verification @@ -95,22 +91,17 @@ data. The default and recommended re-verification interval is 7 days, though an interval as short as 1 day can be set. Shorter intervals reduce risk but increase load and vice versa. -On the **primary** node: +On the **primary** site: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Select **Edit** for the **primary** node to customize the minimum +1. On the left sidebar, select **Geo > Sites**. +1. Select **Edit** for the **primary** site to customize the minimum re-verification interval: ![Re-verification interval](img/reverification-interval.png) The automatic background re-verification is enabled by default, but you can -disable if you need. Run the following commands in a Rails console on the -**primary** node: - -```shell -gitlab-rails console -``` +disable if you need. Run the following commands in a [Rails console](../../operations/rails_console.md) on a **Rails node on the primary** site: To disable automatic background re-verification: @@ -126,11 +117,13 @@ Feature.enable('geo_repository_reverification') ## Reset verification for projects where verification has failed -Geo actively try to correct verification failures marking the repository to +Geo actively tries to correct verification failures marking the repository to be resynced with a back-off period. If you want to reset them manually, this Rake task marks projects where verification has failed or the checksum mismatch to be resynced without the back-off period: +Run the appropriate commands on a **Rails node on the primary** site. + For repositories: ```shell @@ -145,9 +138,9 @@ sudo gitlab-rake geo:verification:wiki:reset ## Reconcile differences with checksum mismatches -If the **primary** and **secondary** nodes have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: +If the **primary** and **secondary** sites have a checksum verification mismatch, the cause may not be apparent. To find the cause of a checksum mismatch: -1. On the **primary** node: +1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Projects**. 1. Find the project that you want to check the checksum differences and @@ -157,31 +150,32 @@ If the **primary** and **secondary** nodes have a checksum verification mismatch ![Project administration page](img/checksum-differences-admin-project-page.png) -1. Go to the project's repository directory on both **primary** and **secondary** nodes - (the path is usually `/var/opt/gitlab/git-data/repositories`). If `git_data_dirs` +1. On a **Gitaly node on the primary** site and a **Gitaly node on the secondary** site, go to the project's repository directory. If using Gitaly Cluster, [check that it is in a healthy state](../../gitaly/troubleshooting.md#check-cluster-health) prior to running these commands. + + The default path is `/var/opt/gitlab/git-data/repositories`. If `git_data_dirs` is customized, check the directory layout on your server to be sure: ```shell cd /var/opt/gitlab/git-data/repositories ``` -1. Run the following command on the **primary** node, redirecting the output to a file: + 1. Run the following command on the **primary** site, redirecting the output to a file: - ```shell - git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > primary-node-refs - ``` + ```shell + git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > primary-site-refs + ``` -1. Run the following command on the **secondary** node, redirecting the output to a file: + 1. Run the following command on the **secondary** site, redirecting the output to a file: - ```shell - git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > secondary-node-refs - ``` + ```shell + git show-ref --head | grep -E "HEAD|(refs/(heads|tags|keep-around|merge-requests|environments|notes)/)" > secondary-site-refs + ``` -1. Copy the files from the previous steps on the same system, and do a diff between the contents: + 1. Copy the files from the previous steps on the same system, and do a diff between the contents: - ```shell - diff primary-node-refs secondary-node-refs - ``` + ```shell + diff primary-site-refs secondary-site-refs + ``` ## Current limitations @@ -191,10 +185,10 @@ progress to include them in [Geo: Verify all replicated data](https://gitlab.com For now, you can verify their integrity manually by following [these instructions](../../raketasks/check.md) on both -nodes, and comparing the output between them. +sites, and comparing the output between them. In GitLab EE 12.1, Geo calculates checksums for attachments, LFS objects, and -archived traces on secondary nodes after the transfer, compares it with the +archived traces on secondary sites after the transfer, compares it with the stored checksums, and rejects transfers if mismatched. Geo currently does not support an automatic way to verify these data if they have been synced before GitLab EE 12.1. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index e5b9a14cfbd..f457cb4b0a2 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -413,7 +413,7 @@ required: 1. Promote the replica database associated with the **secondary** site. This sets the database to read-write. The instructions vary depending on where your database is hosted: - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) - - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-read-replicas-portal#stop-replication) - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) - For other external PostgreSQL databases, save the following script in your secondary site, for example `/tmp/geo_promote.sh`, and modify the connection diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index c351b4031b5..5a5d896c20a 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -60,8 +60,8 @@ Alternatively, you can [back up](../../../raketasks/backup_restore.md#back-up-gi the container registry on the primary site and restore it onto the secondary site: -1. On your primary site, back up only the registry and [exclude specific directories -from the backup](../../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup): +1. On your primary site, back up only the registry and + [exclude specific directories from the backup](../../../raketasks/backup_gitlab.md#excluding-specific-directories-from-the-backup): ```shell # Create a backup in the /var/opt/gitlab/backups folder diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index c5472e412d3..cf7d2649142 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -123,7 +123,7 @@ The following are required to run Geo: - PostgreSQL 13 is not supported for Geo, see [epic 3832](https://gitlab.com/groups/gitlab-org/-/epics/3832) - Git 2.9 or later - Git-lfs 2.4.2 or later on the user side when using LFS -- All sites must run the same GitLab version. +- All sites must run [the same GitLab and PostgreSQL versions](setup/database.md#postgresql-replication). Additionally, check the GitLab [minimum requirements](../../install/requirements.md), and we recommend you use the latest version of GitLab for a better experience. diff --git a/doc/administration/geo/replication/configuration.md b/doc/administration/geo/replication/configuration.md index 043b96478c9..7666a450648 100644 --- a/doc/administration/geo/replication/configuration.md +++ b/doc/administration/geo/replication/configuration.md @@ -12,7 +12,9 @@ type: howto NOTE: This is the final step in setting up a **secondary** Geo site. Stages of the setup process must be completed in the documented order. -Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). +If not, [complete all prior stages](../setup/index.md#using-omnibus-gitlab) before procceed. + +Make sure you [set up the database replication](../setup/database.md), and [configured fast lookup of authorized SSH keys](../../operations/fast_ssh_key_lookup.md) in **both primary and secondary sites**. The basic steps of configuring a **secondary** site are to: @@ -318,7 +320,7 @@ the **primary** site. After you sign in: 1. Verify that it's correctly identified as a **secondary** Geo site, and that Geo is enabled. -The initial replication, or 'backfill', is probably still in progress. You +The initial replication may take some time. The status of the site or the ‘backfill’ may still in progress. You can monitor the synchronization process on each Geo site from the **primary** site's **Geo Sites** dashboard in your browser. diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index e49d5260233..acd27d5feed 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -202,8 +202,8 @@ successfully, you must replicate their data using some other means. |[External merge request diffs](../../merge_request_diffs.md) | **Yes** (13.5) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Replication is behind the feature flag `geo_merge_request_diff_replication`, enabled by default. Verification was behind the feature flag `geo_merge_request_diff_verification`, removed in 14.7.| |[Versioned snippets](../../../user/snippets.md#versioned-snippets) | [**Yes** (13.7)](https://gitlab.com/groups/gitlab-org/-/epics/2809) | [**Yes** (14.2)](https://gitlab.com/groups/gitlab-org/-/epics/2810) | N/A | N/A | Verification was implemented behind the feature flag `geo_snippet_repository_verification` in 13.11, and the feature flag was removed in 14.2. | |[GitLab Pages](../../pages/index.md) | [**Yes** (14.3)](https://gitlab.com/groups/gitlab-org/-/epics/589) | **Yes** (14.6) | [**Yes** (15.1)](https://gitlab.com/groups/gitlab-org/-/epics/5551) | [No](object_storage.md#verification-of-files-in-object-storage) | Behind feature flag `geo_pages_deployment_replication`, enabled by default. Verification was behind the feature flag `geo_pages_deployment_verification`, removed in 14.7. | -|[Incident Metric Images](../../../operations/incident_management/incidents.md#metrics) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/352326) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | No | | -|[Alert Metric Images](../../../operations/incident_management/alerts.md#metrics-tab) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/352326) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | No | | +|[Incident Metric Images](../../../operations/incident_management/incidents.md#metrics) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362561) | No | No | | +|[Alert Metric Images](../../../operations/incident_management/alerts.md#metrics-tab) | [Planned](https://gitlab.com/gitlab-org/gitlab/-/issues/362564) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/362564) | No | No | | |[Server-side Git hooks](../../server_hooks.md) | [Not planned](https://gitlab.com/groups/gitlab-org/-/epics/1867) | No | N/A | N/A | Not planned because of current implementation complexity, low customer interest, and availability of alternatives to hooks. | |[Elasticsearch integration](../../../integration/elasticsearch.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/1186) | No | No | No | Not planned because further product discovery is required and Elasticsearch (ES) clusters can be rebuilt. Secondaries use the same ES cluster as the primary. | |[Dependency proxy images](../../../user/packages/dependency_proxy/index.md) | [Not planned](https://gitlab.com/gitlab-org/gitlab/-/issues/259694) | No | No | No | Blocked by [Geo: Secondary Mimicry](https://gitlab.com/groups/gitlab-org/-/epics/1528). Replication of this cache is not needed for disaster recovery purposes because it can be recreated from external sources. | diff --git a/doc/administration/geo/replication/object_storage.md b/doc/administration/geo/replication/object_storage.md index dd1f982b3a1..d2e10678f8c 100644 --- a/doc/administration/geo/replication/object_storage.md +++ b/doc/administration/geo/replication/object_storage.md @@ -34,8 +34,7 @@ See [Object storage replication tests](geo_validation_tests.md#object-storage-re ## Enabling GitLab-managed object storage replication -> The beta feature was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10586) in GitLab 12.4. -> The feature was made [generally available] in GitLab 15.1. +> The feature was made [generally available](https://gitlab.com/groups/gitlab-org/-/epics/5551) in GitLab 15.1. **Secondary** sites can replicate files stored on the **primary** site regardless of whether they are stored on the local file system or in object storage. @@ -79,7 +78,7 @@ the bucket used by **secondary** sites. If you are using Google Cloud Storage, consider using [Multi-Regional Storage](https://cloud.google.com/storage/docs/storage-classes#multi-regional). -Or you can use the [Storage Transfer Service](https://cloud.google.com/storage-transfer/docs/), +Or you can use the [Storage Transfer Service](https://cloud.google.com/storage-transfer/docs/overview), although this only supports daily synchronization. For manual synchronization, or scheduled by `cron`, see: diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index bb7dbef214b..082ecbbb208 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -183,7 +183,7 @@ This machine's Geo node name matches a database record ... no ``` Learn more about recommended site names in the description of the Name field in -[Geo Admin Area Common Settings](../../../user/admin_area/geo_nodes.md#common-settings). +[Geo Admin Area Common Settings](../../../user/admin_area/geo_sites.md#common-settings). ### Message: `WARNING: oldest xmin is far in the past` and `pg_wal` size growing @@ -519,7 +519,7 @@ To solve this: curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<first_failed_geo_sync_ID>" ``` -1. Enter the [Rails console](../../troubleshooting/navigating_gitlab_via_rails_console.md) and run: +1. Enter the [Rails console](../../operations/rails_console.md) and run: ```ruby failed_geo_syncs = Geo::ProjectRegistry.failed.pluck(:id) @@ -651,8 +651,11 @@ to start again from scratch, there are a few steps that can help you: 1. Reset the Tracking Database. + WARNING: + If you skipped the optional step 3, be sure both `geo-postgresql` and `postgresql` services are running. + ```shell - gitlab-rake db:drop:geo # on a secondary app node + gitlab-rake db:drop:geo DISABLE_DATABASE_ENVIRONMENT_CHECK=1 # on a secondary app node gitlab-ctl reconfigure # on the tracking database node gitlab-rake db:migrate:geo # on a secondary app node ``` @@ -805,7 +808,7 @@ You can work around this by marking the objects as synced and succeeded verifica be aware that can also mark objects that may be [missing from the primary](#missing-files-on-the-geo-primary-site). -To do that, enter the [Rails console](../../troubleshooting/navigating_gitlab_via_rails_console.md) +To do that, enter the [Rails console](../../operations/rails_console.md) and run: ```ruby @@ -817,16 +820,16 @@ end ### Message: curl 18 transfer closed with outstanding read data remaining & fetch-pack: unexpected disconnect while reading sideband packet -Unstable networking conditions can cause Gitaly to fail when trying to fetch large repository +Unstable networking conditions can cause Gitaly to fail when trying to fetch large repository data from the primary site. This is more likely to happen if a repository has to be replicated from scratch between sites. Geo retries several times, but if the transmission is consistently interrupted -by network hiccups, an alternative method such as `rsync` can be used to circumvent `git` and +by network hiccups, an alternative method such as `rsync` can be used to circumvent `git` and create the initial copy of any repository that fails to be replicated by Geo. We recommend transferring each failing repository individually and checking for consistency -after each transfer. Follow the [single target `rsync` instructions](../../operations/moving_repositories.md#single-rsync-to-another-server) +after each transfer. Follow the [single target `rsync` instructions](../../operations/moving_repositories.md#single-rsync-to-another-server) to transfer each affected repository from the primary to the secondary site. ## Fixing errors during a failover or when promoting a secondary to a primary node diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md index 9a1aab8c238..e8c290e197b 100644 --- a/doc/administration/geo/secondary_proxy/index.md +++ b/doc/administration/geo/secondary_proxy/index.md @@ -124,7 +124,7 @@ sudo gitlab-rails runner "Feature.disable(:geo_secondary_proxy_separate_urls)" ``` In Kubernetes, you can run the same command in the toolbox pod. Refer to the -[Kubernetes cheat sheet](../../troubleshooting/kubernetes_cheat_sheet.md#gitlab-specific-kubernetes-information) +[Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html#gitlab-specific-kubernetes-information) for details. ## Limitations diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index 7dfca4c8f73..c0ed7829fce 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -17,8 +17,11 @@ PostgreSQL instances, the Omnibus roles cannot perform all necessary configuration steps. In this case, use the [Geo with external PostgreSQL instances](external_database.md) process instead. +NOTE: The stages of the setup process must be completed in the documented order. -Before you attempt the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). +If not, [complete all prior stages](../setup/index.md#using-omnibus-gitlab) before proceeding. + +Ensure the **secondary** site is running the same version of GitLab Enterprise Edition as the **primary** site. Confirm you have added the [premium or higher licenses](https://about.gitlab.com/pricing/) to your **primary** site. Be sure to read and review all of these steps before you execute them in your testing or production environments. @@ -52,8 +55,9 @@ The following guide assumes that: which includes the [`pg_basebackup` tool](https://www.postgresql.org/docs/12/app-pgbasebackup.html). - You have a **primary** node already set up (the GitLab server you are replicating from), running Omnibus' PostgreSQL (or equivalent version), and - you have a new **secondary** server set up with the same versions of the OS, - PostgreSQL, and GitLab on all nodes. + you have a new **secondary** server set up with the same + [versions of PostgreSQL](../index.md#requirements-for-running-geo), + OS, and GitLab on all nodes. WARNING: Geo works with streaming replication. Logical replication is not supported at this time. @@ -478,7 +482,7 @@ data before running `pg_basebackup`. ```shell gitlab-ctl replicate-geo-database \ --slot-name=<secondary_node_name> \ - --host=<primary_node_ip> + --host=<primary_node_ip> \ --sslmode=verify-ca ``` diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 7e8ffa829f9..b87baa658a1 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -10,14 +10,19 @@ This document is relevant if you are using a PostgreSQL instance that is *not managed by Omnibus*. This includes cloud-managed instances like Amazon RDS, or manually installed and configured PostgreSQL instances. +Ensure that you are using one of the PostgreSQL versions that +[Omnibus ships with](../../package_information/postgresql_versions.md) +to [avoid version mismatches](../index.md#requirements-for-running-geo) +in case a Geo site has to be rebuilt. + NOTE: We strongly recommend running Omnibus-managed instances as they are actively developed and tested. We aim to be compatible with most external (not managed by Omnibus) databases but we do not guarantee compatibility. -## **Primary** node +## **Primary** site -1. SSH into a GitLab **primary** application server and login as root: +1. SSH into a **Rails node on your primary** site and login as root: ```shell sudo -i @@ -39,13 +44,13 @@ developed and tested. We aim to be compatible with most external gitlab_rails['geo_node_name'] = '<site_name_here>' ``` -1. Reconfigure the **primary** node for the change to take effect: +1. Reconfigure the **Rails node** for the change to take effect: ```shell gitlab-ctl reconfigure ``` -1. Execute the command below to define the node as **primary** node: +1. Execute the command below on the **Rails node** to define the site as **primary** site: ```shell gitlab-ctl set-geo-primary-node @@ -62,10 +67,10 @@ To set up an external database, you can either: #### Leverage your cloud provider's tools to replicate the primary database -Given you have a primary node set up on AWS EC2 that uses RDS. +Given you have a primary site set up on AWS EC2 that uses RDS. You can now just create a read-only replica in a different region and the replication process is managed by AWS. Make sure you've set Network ACL (Access Control List), Subnet, and -Security Group according to your needs, so the secondary application node can access the database. +Security Group according to your needs, so the secondary Rails node(s) can access the database. The following instructions detail how to create a read-only replica for common cloud providers: @@ -74,7 +79,7 @@ cloud providers: - Azure Database for PostgreSQL - [Create and manage read replicas in Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-read-replicas-portal) - Google Cloud SQL - [Creating read replicas](https://cloud.google.com/sql/docs/postgres/replication/create-replica) -Once your read-only replica is set up, you can skip to [configure your secondary application node](#configure-secondary-application-nodes-to-use-the-external-read-replica). +Once your read-only replica is set up, you can skip to [configure your secondary site](#configure-secondary-site-to-use-the-external-read-replica) #### Manually configure the primary database for replication @@ -107,7 +112,7 @@ max_replication_slots = 1 # number of secondary instances hot_standby = on ``` -## **Secondary** nodes +## **Secondary** sites ### Manually configure the replica database @@ -136,7 +141,7 @@ wal_keep_segments = 10 hot_standby = on ``` -### Configure **secondary** application nodes to use the external read-replica +### Configure **secondary** site to use the external read-replica With Omnibus, the [`geo_secondary_role`](https://docs.gitlab.com/omnibus/roles/#gitlab-geo-roles) @@ -148,7 +153,7 @@ has three main functions: To configure the connection to the external read-replica database and enable Log Cursor: -1. SSH into a GitLab **secondary** application server and login as root: +1. SSH into each **Rails, Sidekiq and Geo Log Cursor** node on your **secondary** site and login as root: ```shell sudo -i @@ -179,7 +184,7 @@ To configure the connection to the external read-replica database and enable Log ### Configure the tracking database -**Secondary** nodes use a separate PostgreSQL installation as a tracking +**Secondary** sites use a separate PostgreSQL installation as a tracking database to keep track of replication status and automatically recover from potential replication issues. Omnibus automatically configures a tracking database when `roles ['geo_secondary_role']` is set. @@ -209,7 +214,7 @@ the tracking database on port 5432. [database requirements document](../../../install/requirements.md#database). 1. Set up a `gitlab_geo` user with a password of your choice, create the `gitlabhq_geo_production` database, and make the user an owner of the database. You can see an example of this setup in the [installation from source documentation](../../../install/installation.md#6-database). 1. If you are **not** using a cloud-managed PostgreSQL database, ensure that your secondary - node can communicate with your tracking database by manually changing the + site can communicate with your tracking database by manually changing the `pg_hba.conf` that is associated with your tracking database. Remember to restart PostgreSQL afterwards for the changes to take effect: diff --git a/doc/administration/geo/setup/index.md b/doc/administration/geo/setup/index.md index f4c21293782..5ddfee6774e 100644 --- a/doc/administration/geo/setup/index.md +++ b/doc/administration/geo/setup/index.md @@ -12,21 +12,25 @@ These instructions assume you have a working instance of GitLab. They guide you 1. Making your existing instance the **primary** site. 1. Adding **secondary** sites. +You must use a [GitLab Premium](https://about.gitlab.com/pricing/) license or higher, +but you only need one license for all the sites. + WARNING: -The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all sites.** +The steps below should be followed in the order they appear. **Make sure the GitLab version is the same on all sites. Do not create an account or log in to the new secondary.** ## Using Omnibus GitLab If you installed GitLab using the Omnibus packages (highly recommended): -1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that serve as the **secondary** site. Do not create an account or log in to the new **secondary** site. The **GitLab version must match** across primary and secondary sites. +1. [Install GitLab Enterprise Edition](https://about.gitlab.com/install/) on the nodes that serve as the **secondary** site. **Do not create an account or log in** to the new **secondary** site. The **GitLab version must match** across primary and secondary sites. 1. [Add the GitLab License](../../../user/admin_area/license.md) on the **primary** site to unlock Geo. The license must be for [GitLab Premium](https://about.gitlab.com/pricing/) or higher. 1. [Set up the database replication](database.md) (`primary (read-write) <-> secondary (read-only)` topology). 1. [Configure fast lookup of authorized SSH keys in the database](../../operations/fast_ssh_key_lookup.md). This step is required and needs to be done on **both** the **primary** and **secondary** sites. 1. [Configure GitLab](../replication/configuration.md) to set the **primary** and **secondary** sites. +1. Optional: [Configure Object storage](../../object_storage.md) 1. Optional: [Configure a secondary LDAP server](../../auth/ldap/index.md) for the **secondary** sites. See [notes on LDAP](../index.md#ldap). +1. Optional: [Configure Geo secondary proxying](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. This step is recommended to accelerate most read requests while transparently proxying writes to the primary Geo site. 1. Follow the [Using a Geo Site](../replication/usage.md) guide. -1. [Configure Geo secondary proxying](../secondary_proxy/index.md) to use a single, unified URL for all Geo sites. This step is recommended to accelerate most read requests while transparently proxying writes to the primary Geo site. ## Post-installation documentation diff --git a/doc/administration/gitaly/configure_gitaly.md b/doc/administration/gitaly/configure_gitaly.md index 426d07b154d..4b2832bebc0 100644 --- a/doc/administration/gitaly/configure_gitaly.md +++ b/doc/administration/gitaly/configure_gitaly.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -130,57 +130,49 @@ Install Gitaly on each Gitaly server using either Omnibus GitLab or install it f - To install from source, follow the steps at [Install Gitaly](../../install/installation.md#install-gitaly). -### Configure authentication +### Configure Gitaly servers -Gitaly and GitLab use two shared secrets for authentication: +To configure Gitaly servers, you must: -- One to authenticate gRPC requests to Gitaly. -- A second for authentication callbacks from GitLab Shell to the GitLab internal API. +- Configure authentication. +- Configure storage paths. +- Enable the network listener. -**For Omnibus GitLab** +The `git` user must be able to read, write, and set permissions on the configured storage path. -To configure the Gitaly token: +To avoid downtime while rotating Gitaly's token, you can temporarily disable authentication using the `gitaly['auth_transitioning']` setting. For more information, see the documentation on +[enabling "auth transitioning mode"](#enable-auth-transitioning-mode). -1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: +#### Configure authentication - ```ruby - gitlab_rails['gitaly_token'] = 'abc123secret' - ``` +Gitaly and GitLab use two shared secrets for authentication: -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. On the Gitaly server, edit `/etc/gitlab/gitlab.rb`: +- _Gitaly token_: used to authenticate gRPC requests to Gitaly +- _GitLab Shell token_: used for authentication callbacks from GitLab Shell to the GitLab internal API + +**For Omnibus GitLab** + +To configure the _Gitaly token_, edit `/etc/gitlab/gitlab.rb`: ```ruby gitaly['auth_token'] = 'abc123secret' ``` -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - -There are two ways to configure the GitLab Shell token. +There are two ways to configure the _GitLab Shell token_. -Method 1: +Method 1 (recommended): -1. Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly servers +Copy `/etc/gitlab/gitlab-secrets.json` from the Gitaly client to same path on the Gitaly servers (and any other Gitaly clients). -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) on Gitaly servers. Method 2: -1. On the Gitaly clients, edit `/etc/gitlab/gitlab.rb`: +Edit `/etc/gitlab/gitlab.rb`: ```ruby gitlab_shell['secret_token'] = 'shellsecret' ``` -1. Save the file and [reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). -1. On the Gitaly servers, edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_shell['secret_token'] = 'shellsecret' - ``` - -1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). - **For installations from source** 1. Copy `/home/git/gitlab/.gitlab_shell_secret` from the Gitaly client to the same path on the @@ -203,14 +195,7 @@ Method 2: 1. Save the file and [restart GitLab](../restart_gitlab.md#installations-from-source). -### Configure Gitaly servers - -On the Gitaly servers, you must configure storage paths and enable the network listener. -The Gitaly server must be able to read, write, and set permissions on the configured path. - -If you want to reduce the risk of downtime when you enable authentication, you can temporarily -disable enforcement. For more information, see the documentation on configuring -[Gitaly authentication](https://gitlab.com/gitlab-org/gitaly/blob/master/doc/configuration/README.md#authentication). +#### Configure Gitaly server **For Omnibus GitLab** @@ -904,7 +889,7 @@ gitaly['cgroups_repositories_cpu_shares'] => 512 which represents 100% of CPU. This value cannot exceed that of the top level`cgroups_cpu_shares`. -#### Configure cgroups (legacy method) +#### Configure cgroups (legacy method) To configure cgroups in Gitaly for GitLab versions using the legacy method, add `gitaly['cgroups']` to `/etc/gitlab/gitlab.rb`. For example: diff --git a/doc/administration/gitaly/index.md b/doc/administration/gitaly/index.md index 160313073a5..c543f62f135 100644 --- a/doc/administration/gitaly/index.md +++ b/doc/administration/gitaly/index.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -71,7 +71,7 @@ the current status of these issues, please refer to the referenced issues and ep | Issue | Summary | How to avoid | |:--------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|:--------------------------------| | Gitaly Cluster + Geo - Issues retrying failed syncs | If Gitaly Cluster is used on a Geo secondary site, repositories that have failed to sync could continue to fail when Geo tries to resync them. Recovering from this state requires assistance from support to run manual steps. Work is in-progress to update Gitaly Cluster to [identify repositories with a unique and persistent identifier](https://gitlab.com/gitlab-org/gitaly/-/issues/3485), which is expected to resolve the issue. | No known solution at this time. | -| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting live upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. | +| Praefect unable to insert data into the database due to migrations not being applied after an upgrade | If the database is not kept up to date with completed migrations, then the Praefect node is unable to perform normal operation. | Make sure the Praefect database is up and running with all migrations completed (For example: `/opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status` should show a list of all applied migrations). Consider [requesting upgrade assistance](https://about.gitlab.com/support/scheduling-upgrade-assistance/) so your upgrade plan can be reviewed by support. | | Restoring a Gitaly Cluster node from a snapshot in a running cluster | Because the Gitaly Cluster runs with consistent state, introducing a single node that is behind will result in the cluster not being able to reconcile the nodes data and other nodes data | Don't restore a single Gitaly Cluster node from a backup snapshot. If you must restore from backup, it's best to snapshot all Gitaly Cluster nodes at the same time and take a database dump of the Praefect database. | ### Snapshot backup and recovery limitations @@ -523,6 +523,48 @@ For more information on configuring Gitaly Cluster, see [Configure Gitaly Cluste To upgrade a Gitaly Cluster, follow the documentation for [zero downtime upgrades](../../update/zero_downtime.md#gitaly-or-gitaly-cluster). +### Downgrade Gitaly Cluster to a previous version + +If you need to roll back a Gitaly Cluster to an earlier version, some Praefect database migrations may need to be reverted. In a cluster with: + +- A single Praefect node, this happens when GitLab itself is downgraded. +- Multiple Praefect nodes, additional steps are required. + +To downgrade a Gitaly Cluster with multiple Praefect nodes: + +1. Stop the Praefect service on all Praefect nodes: + + ```shell + gitlab-ctl stop praefect + ``` + +1. Downgrade the GitLab package to the older version on one of the Praefect nodes. +1. On the downgraded node, check the state of Praefect migrations: + + ```shell + /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate-status + ``` + +1. Count the number of migrations with `unknown migration` in the `APPLIED` column. +1. On a Praefect node that has **not** been downgraded, perform a dry run of the rollback to validate which migrations to revert. `<CT_UNKNOWN>` + is the number of unknown migrations reported by the downgraded node. + + ```shell + /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate <CT_UNKNOWN> + ``` + +1. If the results look correct, run the same command with the `-f` option to revert the migrations: + + ```shell + /opt/gitlab/embedded/bin/praefect -config /var/opt/gitlab/praefect/config.toml sql-migrate -f <CT_UNKNOWN> + ``` + +1. Downgrade the GitLab package on the remaining Praefect nodes and start the Praefect service again: + + ```shell + gitlab-ctl start praefect + ``` + ## Migrate to Gitaly Cluster WARNING: diff --git a/doc/administration/gitaly/monitoring.md b/doc/administration/gitaly/monitoring.md index ac0c4cf139d..0dd9e0d7128 100644 --- a/doc/administration/gitaly/monitoring.md +++ b/doc/administration/gitaly/monitoring.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index f6166d713b1..5635898293b 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -833,6 +833,7 @@ For more information on Gitaly server configuration, see our sidekiq['enable'] = false gitlab_workhorse['enable'] = false prometheus_monitoring['enable'] = false + gitlab_kas['enable'] = false # Enable only the Gitaly service gitaly['enable'] = true diff --git a/doc/administration/gitaly/recovery.md b/doc/administration/gitaly/recovery.md index 2b5916be916..551a7ab289a 100644 --- a/doc/administration/gitaly/recovery.md +++ b/doc/administration/gitaly/recovery.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/gitaly/reference.md b/doc/administration/gitaly/reference.md index d1e802111cd..91780ec5661 100644 --- a/doc/administration/gitaly/reference.md +++ b/doc/administration/gitaly/reference.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index a1f542ddac8..fb3a31d61b8 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -1,5 +1,5 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -220,6 +220,12 @@ on the Gitaly server matches the one on Gitaly client. If it doesn't match, update the secrets file on the Gitaly server to match the Gitaly client, then [reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure). +If you've confirmed that your `gitlab-secrets.json` file is the same on all Gitaly servers and clients, +the application might be fetching this secret from a different file. Your Gitaly server's +`config.toml file` indicates the secrets file in use. +If that setting is missing, GitLab defaults to using `.gitlab_shell_secret` under +`/opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret`. + ### Repository pushes fail with a `deny updating a hidden ref` error Due to [a change](https://gitlab.com/gitlab-org/gitaly/-/merge_requests/3426) diff --git a/doc/administration/housekeeping.md b/doc/administration/housekeeping.md index 93409b54c0d..31b7ac9fd3e 100644 --- a/doc/administration/housekeeping.md +++ b/doc/administration/housekeeping.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Housekeeping **(FREE SELF)** -GitLab supports and automates housekeeping tasks within your current repository such as: +GitLab supports and automates housekeeping tasks in your current repository such as: - Compressing Git objects. - Removing unreachable objects. diff --git a/doc/administration/index.md b/doc/administration/index.md index 29fb65e2deb..4565c0a8e00 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -207,15 +207,14 @@ Learn how to install, configure, update, and maintain your GitLab instance. ## Troubleshooting -- [Debugging tips](troubleshooting/debug.md): Tips to debug problems when things go wrong. - [Log system](logs.md): Where to look for logs. - [Sidekiq Troubleshooting](troubleshooting/sidekiq.md): Debug when Sidekiq appears hung and is not processing jobs. - [Troubleshooting Elasticsearch](troubleshooting/elasticsearch.md) -- [Navigating GitLab via Rails console](troubleshooting/navigating_gitlab_via_rails_console.md) +- [Navigating GitLab via Rails console](operations/rails_console.md) - [GitLab application limits](instance_limits.md) - [Responding to security incidents](../security/responding_to_security_incidents.md) -### Support Team Docs +### Support Team Documentation The GitLab Support Team has collected a lot of information about troubleshooting GitLab. The following documents are used by the Support Team or by customers @@ -231,7 +230,7 @@ who are aware of the risks. - [Diagnostics tools](troubleshooting/diagnostics_tools.md) - [Linux commands](troubleshooting/linux_cheat_sheet.md) -- [Troubleshooting Kubernetes](troubleshooting/kubernetes_cheat_sheet.md) +- [Troubleshooting Kubernetes](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) - [Troubleshooting PostgreSQL](troubleshooting/postgresql.md) - [Guide to test environments](troubleshooting/test_environments.md) (for Support Engineers) - [GitLab Rails console commands](troubleshooting/gitlab_rails_cheat_sheet.md) (for Support Engineers) diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index d86414ae285..2007f5edb3b 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -7,7 +7,7 @@ type: reference # GitLab application limits **(FREE SELF)** -GitLab, like most large applications, enforces limits within certain features to maintain a +GitLab, like most large applications, enforces limits in certain features to maintain a minimum quality of performance. Allowing some features to be limitless could affect security, performance, data, or could even exhaust the allocated resources for the application. @@ -156,7 +156,7 @@ Set the limit to `0` to disable it. ### Search rate limit -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80631) in GitLab 14.9 +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/80631) in GitLab 14.9. This setting limits global search requests. @@ -309,7 +309,7 @@ Blocked recursive webhook calls are logged in `auth.log` with the message `"Recu The [minimum wait time between pull refreshes](../user/project/repository/mirror/index.md) defaults to 300 seconds (5 minutes). For example, a pull refresh only runs once in a given 300 second period, regardless of how many times you trigger it. -This setting applies in the context of pull refreshes invoked via the [projects API](../api/projects.md#start-the-pull-mirroring-process-for-a-project), or when forcing an update by selecting the **Update now** (**{retry}**) button within **Settings > Repository > Mirroring repositories**. This setting has no effect on the automatic 30 minute interval schedule used by Sidekiq for [pull mirroring](../user/project/repository/mirror/pull.md). +This setting applies in the context of pull refreshes invoked via the [projects API](../api/projects.md#start-the-pull-mirroring-process-for-a-project), or when forcing an update by selecting **Update now** (**{retry}**) in **Settings > Repository > Mirroring repositories**. This setting has no effect on the automatic 30 minute interval schedule used by Sidekiq for [pull mirroring](../user/project/repository/mirror/pull.md). To change this limit for a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -342,7 +342,7 @@ and to limit memory consumption. When using offset-based pagination in the REST API, there is a limit to the maximum requested offset into the set of results. This limit is only applied to endpoints that support keyset-based pagination. More information about pagination options can be -found in the [API docs section on pagination](../api/index.md#pagination). +found in the [API documentation section on pagination](../api/index.md#pagination). To set this limit for a self-managed installation, run the following in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): @@ -793,19 +793,9 @@ Plan.default.actual_limits.update!(dotenv_size: 5.kilobytes) > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17859) in GitLab 12.5. -You can limit the number of inbound alerts for [incidents](../operations/incident_management/incidents.md) -that can be created in a period of time. The inbound [incident management](../operations/incident_management/index.md) -alert limit can help prevent overloading your incident responders by reducing the -number of alerts or duplicate issues. +This setting limits the number of inbound alert payloads over a period of time. -To set inbound incident management alert limits: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Settings > Network**. -1. Expand General **Incident Management Limits**. -1. Select the **Enable Incident Management inbound alert limit** checkbox. -1. Optional. Input a custom value for **Maximum requests per project per rate limit period**. Default is 3600. -1. Optional. Input a custom value for **Rate limit period**. Default is 3600 seconds. +Read more about [incident management rate limits](../user/admin_area/settings/rate_limit_on_pipelines_creation.md). ### Prometheus Alert JSON payloads @@ -954,7 +944,7 @@ The maximum allowed [push size](../user/admin_area/settings/account_and_limit_se Total number of changes (branches or tags) in a single push. If changes are more than the specified limit, hooks are not executed. -More information can be found in these docs: +More information can be found in these documentations: - [Webhooks push events](../user/project/integrations/webhook_events.md#push-events) - [Project services push hooks limit](../user/project/integrations/index.md#push-hooks-limit) @@ -1050,7 +1040,7 @@ In addition to application-based limits, GitLab.com is configured to use Cloudfl ## Container Repository tag deletion limit -Container repository tags are in the Container Registry and, as such, each tag deletion will trigger network requests to the Container Registry. Because of this, we limit the number of tags that a single API call can delete to 20. +Container repository tags are in the Container Registry and, as such, each tag deletion triggers network requests to the Container Registry. Because of this, we limit the number of tags that a single API call can delete to 20. ## Project-level Secure Files API limits diff --git a/doc/administration/instance_review.md b/doc/administration/instance_review.md index f444589bdf3..d9126036a16 100644 --- a/doc/administration/instance_review.md +++ b/doc/administration/instance_review.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Conversion +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/administration/integration/mailgun.md b/doc/administration/integration/mailgun.md index fbad201be7d..6b0cb0466fc 100644 --- a/doc/administration/integration/mailgun.md +++ b/doc/administration/integration/mailgun.md @@ -23,7 +23,7 @@ After completing the integration, Mailgun `temporary_failure` and `permanent_fai Before you can enable Mailgun in GitLab, set up your own Mailgun endpoints to receive the webhooks. -Using the [Mailgun webhook guide](https://www.mailgun.com/blog/a-guide-to-using-mailguns-webhooks/): +Using the [Mailgun webhook guide](https://www.mailgun.com/blog/product/a-guide-to-using-mailguns-webhooks/): 1. Add a webhook with the **Event type** set to **Permanent Failure**. 1. Enter the URL of your instance and include the `/-/mailgun/webhooks` path. diff --git a/doc/administration/integration/plantuml.md b/doc/administration/integration/plantuml.md index cdf6d48ad41..58c9e01a151 100644 --- a/doc/administration/integration/plantuml.md +++ b/doc/administration/integration/plantuml.md @@ -7,9 +7,11 @@ type: reference, howto # PlantUML and GitLab **(FREE)** -When [PlantUML](https://plantuml.com) integration is enabled and configured in -GitLab, you can create diagrams in snippets, wikis, and repositories. To set up -the integration, you must: +When the [PlantUML](https://plantuml.com) integration is enabled and configured in +GitLab, you can create diagrams in snippets, wikis, and repositories. This integration +is enabled on GitLab.com for all SaaS users and does not require any additional configuration. + +To set up the integration on a self-managed instance, you must: 1. [Configure your PlantUML server](#configure-your-plantuml-server). 1. [Configure local PlantUML access](#configure-local-plantuml-access). diff --git a/doc/administration/invalidate_markdown_cache.md b/doc/administration/invalidate_markdown_cache.md index 855910fec6a..20a77691bbc 100644 --- a/doc/administration/invalidate_markdown_cache.md +++ b/doc/administration/invalidate_markdown_cache.md @@ -5,18 +5,25 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference --- -# Invalidate Markdown Cache **(FREE)** +# Markdown cache **(FREE)** -For performance reasons, GitLab caches the HTML version of Markdown text -in fields like comments, issue descriptions, and merge request descriptions. These -cached versions can become outdated, such as -when the `external_url` configuration option is changed. Links +For performance reasons, GitLab caches the HTML version of Markdown text in fields such as: + +- Comments. +- Issue descriptions. +- Merge request descriptions. + +These cached versions can become outdated, such as when the `external_url` configuration option is changed. Links in the cached text would still refer to the old URL. -To avoid this problem, the administrator can invalidate the existing cache by -increasing the `local_markdown_version` setting in application settings. This can -be done by changing the application settings -[through the API](../api/settings.md#change-application-settings): +## Invalidate the cache + +Pre-requisite: + +- You must be an administrator. + +To avoid problems caused by cached HTML versions, invalidate the existing cache by increasing the `local_markdown_version` +setting in application settings [using the API](../api/settings.md#change-application-settings): ```shell curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/application/settings?local_markdown_version=<increased_number>" diff --git a/doc/administration/lfs/index.md b/doc/administration/lfs/index.md index ebf76cc6aec..2fcf7d2f276 100644 --- a/doc/administration/lfs/index.md +++ b/doc/administration/lfs/index.md @@ -184,7 +184,11 @@ Verify `objectstg` below (where `store=2`) has count of all LFS objects: ```shell gitlabhq_production=# SELECT count(*) AS total, sum(case when file_store = '1' then 1 else 0 end) AS filesystem, sum(case when file_store = '2' then 1 else 0 end) AS objectstg FROM lfs_objects; +``` + +**Example Output** +```shell total | filesystem | objectstg ------+------------+----------- 2409 | 0 | 2409 diff --git a/doc/administration/libravatar.md b/doc/administration/libravatar.md index 442c233a536..eac7c6f848b 100644 --- a/doc/administration/libravatar.md +++ b/doc/administration/libravatar.md @@ -12,7 +12,7 @@ GitLab by default supports the [Gravatar](https://gravatar.com) avatar service. Libravatar is another service that delivers your avatar (profile picture) to other websites. The Libravatar API is [heavily based on gravatar](https://wiki.libravatar.org/api/), so you can -easily switch to the Libravatar avatar service or even your own Libravatar +switch to the Libravatar avatar service or even your own Libravatar server. ## Configuration @@ -45,7 +45,7 @@ the URL is different in the configuration, but you must provide the same placeholders so GitLab can parse the URL correctly. For example, you host a service on `http://libravatar.example.com` and the -`plain_url` you need to supply in `gitlab.yml` is +`plain_url` you must supply in `gitlab.yml` is `http://libravatar.example.com/avatar/%{hash}?s=%{size}&d=identicon` diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index e6a1c0c25d5..988bddcfe62 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -11,8 +11,8 @@ traffic to the application servers. The specifics on which load balancer to use or the exact configuration is beyond the scope of GitLab documentation. We hope that if you're managing HA systems like GitLab you have a load balancer of choice already. Some examples including HAProxy (open-source), F5 Big-IP LTM, -and Citrix Net Scaler. This documentation outlines what ports and protocols -you need to use with GitLab. +and Citrix NetScaler. This documentation outlines what ports and protocols +to use with GitLab. ## SSL @@ -40,7 +40,7 @@ Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. The load balancers is be responsible for managing SSL certificates and terminating SSL. -Since communication between the load balancers and GitLab isn't secure, +Because communication between the load balancers and GitLab isn't secure, there is some additional configuration needed. See [NGINX Proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) for details. @@ -52,7 +52,7 @@ The load balancers is responsible for managing SSL certificates that end users see. Traffic is secure between the load balancers and NGINX in this -scenario. There is no need to add configuration for proxied SSL since the +scenario. There is no need to add configuration for proxied SSL because the connection is secure all the way. However, configuration must be added to GitLab to configure SSL certificates. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) diff --git a/doc/administration/logs.md b/doc/administration/logs.md index 5a2c1190896..671c264ed85 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -1063,23 +1063,23 @@ For Omnibus GitLab installations, Mattermost logs are in these locations: ## Workhorse Logs -For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/`. +For Omnibus GitLab installations, Workhorse logs are in `/var/log/gitlab/gitlab-workhorse/current`. ## PostgreSQL Logs -For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/`. +For Omnibus GitLab installations, PostgreSQL logs are in `/var/log/gitlab/postgresql/current`. ## Prometheus Logs -For Omnibus GitLab installations, Prometheus logs are in `/var/log/gitlab/prometheus/`. +For Omnibus GitLab installations, Prometheus logs are in `/var/log/gitlab/prometheus/current`. ## Redis Logs -For Omnibus GitLab installations, Redis logs are in `/var/log/gitlab/redis/`. +For Omnibus GitLab installations, Redis logs are in `/var/log/gitlab/redis/current`. ## Alertmanager Logs -For Omnibus GitLab installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/`. +For Omnibus GitLab installations, Alertmanager logs are in `/var/log/gitlab/alertmanager/current`. <!-- vale gitlab.Spelling = NO --> @@ -1091,11 +1091,11 @@ For Omnibus GitLab installations, crond logs are in `/var/log/gitlab/crond/`. ## Grafana Logs -For Omnibus GitLab installations, Grafana logs are in `/var/log/gitlab/grafana/`. +For Omnibus GitLab installations, Grafana logs are in `/var/log/gitlab/grafana/current`. ## LogRotate Logs -For Omnibus GitLab installations, `logrotate` logs are in `/var/log/gitlab/logrotate/`. +For Omnibus GitLab installations, `logrotate` logs are in `/var/log/gitlab/logrotate/current`. ## GitLab Monitor Logs @@ -1103,12 +1103,12 @@ For Omnibus GitLab installations, GitLab Monitor logs are in `/var/log/gitlab/gi ## GitLab Exporter -For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/`. +For Omnibus GitLab installations, GitLab Exporter logs are in `/var/log/gitlab/gitlab-exporter/current`. ## GitLab agent server For Omnibus GitLab installations, GitLab agent server logs are -in `/var/log/gitlab/gitlab-kas/`. +in `/var/log/gitlab/gitlab-kas/current`. ## Praefect Logs diff --git a/doc/administration/monitoring/performance/grafana_configuration.md b/doc/administration/monitoring/performance/grafana_configuration.md index 79612145327..3a76e2e4578 100644 --- a/doc/administration/monitoring/performance/grafana_configuration.md +++ b/doc/administration/monitoring/performance/grafana_configuration.md @@ -14,7 +14,7 @@ and Grafana allows you to query the data to display useful graphs. Omnibus GitLab can [help you install Grafana (recommended)](https://docs.gitlab.com/omnibus/settings/grafana.html) or Grafana supplies package repositories (Yum/Apt) for easy installation. -See [Grafana installation documentation](https://grafana.com/docs/grafana/latest/installation/) +See [Grafana installation documentation](https://grafana.com/docs/grafana/latest/setup-grafana/installation/) for detailed steps. Before starting Grafana for the first time, set the administration user @@ -133,8 +133,8 @@ However, you should **not** reinstate your old data _except_ under one of the fo If you require access to your old Grafana data but don't meet one of these criteria, you may consider: 1. Reinstating it temporarily. -1. [Exporting the dashboards](https://grafana.com/docs/grafana/latest/reference/export_import/#exporting-a-dashboard) you need. -1. Refreshing the data and [re-importing your dashboards](https://grafana.com/docs/grafana/latest/reference/export_import/#importing-a-dashboard). +1. [Exporting the dashboards](https://grafana.com/docs/grafana/latest/dashboards/export-import/#exporting-a-dashboard) you need. +1. Refreshing the data and [re-importing your dashboards](https://grafana.com/docs/grafana/latest/dashboards/export-import/#import-dashboard). WARNING: These actions pose a temporary vulnerability while your old Grafana data is in use. diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index aefd7a49b8b..759f485c109 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -54,7 +54,7 @@ From left to right, the performance bar displays: - [**First Contentful Paint**](https://web.dev/first-contentful-paint/): Time until something was visible to the user. Displays `NaN` if your browser does not support this feature. - - [**DomContentLoaded**](https://developers.google.com/web/fundamentals/performance/critical-rendering-path/measure-crp) Event. + - [**DomContentLoaded**](https://web.dev/critical-rendering-path-measure-crp/) Event. - **Total number of requests** the page loaded. - **Memory**: the amount of memory consumed and objects allocated during the selected request. Select it to display a window with more details. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 4f8fbd0c07e..d6575766b17 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -42,6 +42,8 @@ The following metrics are available: | `gitlab_ci_pipeline_builder_scoped_variables_duration` | Histogram | 14.5 | Time in seconds it takes to create the scoped variables for a CI/CD job | `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | | `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | +| `gitlab_ci_runner_authentication_success_total` | Counter | 15.2 | Total number of times that runner authentication has succeeded | `type` | +| `gitlab_ci_runner_authentication_failure_total` | Counter | 15.2 | Total number of times that runner authentication has failed | `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | | `job_waiter_timeouts_total` | Counter | 12.9 | Number of batches of jobs that timed out where a web request is waiting for the jobs to complete | `worker` | | `gitlab_ci_active_jobs` | Histogram | 14.2 | Count of active jobs when pipeline is created | | @@ -136,8 +138,8 @@ The following metrics are available: | `pipeline_graph_links_per_job_ratio` | Histogram | 13.9 | Ratio of links to job per graph | | | `gitlab_ci_pipeline_security_orchestration_policy_processing_duration_seconds` | Histogram | 13.12 | Time in seconds it takes to process Security Policies in CI/CD pipeline | | | `gitlab_spamcheck_request_duration_seconds` | Histogram | 13.12 | The duration for requests between Rails and the anti-spam engine | | -| `service_desk_thank_you_email` | Counter | 14.0 | Total number of email responses to new service desk emails | | -| `service_desk_new_note_email` | Counter | 14.0 | Total number of email notifications on new service desk comment | | +| `service_desk_thank_you_email` | Counter | 14.0 | Total number of email responses to new Service Desk emails | | +| `service_desk_new_note_email` | Counter | 14.0 | Total number of email notifications on new Service Desk comment | | | `email_receiver_error` | Counter | 14.1 | Total number of errors when processing incoming emails | | | `gitlab_snowplow_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emitted | | | `gitlab_snowplow_failed_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emission failures | | @@ -176,6 +178,7 @@ configuration option in `gitlab.yml`. These metrics are served from the | `sidekiq_jobs_queue_duration_seconds` | Histogram | 12.5 | Duration in seconds that a Sidekiq job was queued before being executed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_failed_total` | Counter | 12.2 | Sidekiq jobs failed | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_retried_total` | Counter | 12.2 | Sidekiq jobs retried | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | +| `sidekiq_jobs_interrupted_total` | Counter | 15.2 | Sidekiq jobs interrupted | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_jobs_dead_total` | Counter | 13.7 | Sidekiq dead jobs (jobs that have run out of retries) | `queue`, `boundary`, `external_dependencies`, `feature_category`, `urgency` | | `sidekiq_redis_requests_total` | Counter | 13.1 | Redis requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | | `sidekiq_elasticsearch_requests_total` | Counter | 13.1 | Elasticsearch requests during a Sidekiq job execution | `queue`, `boundary`, `external_dependencies`, `feature_category`, `job_status`, `urgency` | @@ -347,6 +350,7 @@ Some basic Ruby runtime metrics are available: |:---------------------------------------- |:--------- |:----- |:----------- | | `ruby_gc_duration_seconds` | Counter | 11.1 | Time spent by Ruby in GC | | `ruby_gc_stat_...` | Gauge | 11.1 | Various metrics from [GC.stat](https://ruby-doc.org/core-2.6.5/GC.html#method-c-stat) | +| `ruby_gc_stat_ext_heap_fragmentation` | Gauge | 15.2 | Degree of Ruby heap fragmentation as live objects versus eden slots (range 0 to 1) | | `ruby_file_descriptors` | Gauge | 11.1 | File descriptors per process | | `ruby_sampler_duration_seconds` | Counter | 11.1 | Time spent collecting stats | | `ruby_process_cpu_seconds_total` | Gauge | 12.0 | Total amount of CPU time per process | diff --git a/doc/administration/monitoring/prometheus/web_exporter.md b/doc/administration/monitoring/prometheus/web_exporter.md index 4b449a1d74e..fe140b15ed2 100644 --- a/doc/administration/monitoring/prometheus/web_exporter.md +++ b/doc/administration/monitoring/prometheus/web_exporter.md @@ -51,3 +51,23 @@ To enable the dedicated server: for the changes to take effect. Metrics can now be served and scraped from `localhost:8083/metrics`. + +## Enable HTTPS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/364771) in GitLab 15.2. + +To serve metrics via HTTPS instead of HTTP, enable TLS in the exporter settings: + +1. Edit `/etc/gitlab/gitlab.rb` to add (or find and uncomment) the following lines: + + ```ruby + puma['exporter_tls_enabled'] = true + puma['exporter_tls_cert_path'] = "/path/to/certificate.pem" + puma['exporter_tls_key_path'] = "/path/to/private-key.pem" + ``` + +1. Save the file and [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +When TLS is enabled, the same `port` and `address` will be used as described above. +The metrics server cannot serve both HTTP and HTTPS at the same time. diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index d9866a6c09f..51fa627b8d4 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -24,7 +24,7 @@ file system performance, see Starting with GitLab version 14.0, support for NFS to store Git repository data is deprecated. Technical customer support and engineering support is available for the 14.x releases. Engineering is fixing bugs and security vulnerabilities consistent with our [release and maintenance policy](../policy/maintenance.md#security-releases). -Upon the release of GitLab 15.6 technical and engineering support for using NFS to store Git repository data will be officially at end-of-life. There will be no product changes or troubleshooting provided via Engineering, Security or Paid Support channels after the release date of 15.6, regardless of your GitLab version. +Upon the release of GitLab 15.6 technical and engineering support for using NFS to store Git repository data is officially at end-of-life. There are no product changes or troubleshooting provided via Engineering, Security or Paid Support channels after the release date of 15.6, regardless of your GitLab version. Until the release of 15.6, for customers running 14.x releases, we continue to help with Git related tickets from customers running one or more Gitaly servers with its data stored on NFS. Examples may include: @@ -268,9 +268,9 @@ version of a directory. From the [Linux man page](https://linux.die.net/man/5/nfs), the important parts: -> If the nocto option is specified, the client uses a non-standard heuristic to determine when files on the server have changed. +> If the `nocto` option is specified, the client uses a non-standard heuristic to determine when files on the server have changed. > -> Using the nocto option may improve performance for read-only mounts, but should be used only if the data on the server changes only occasionally. +> Using the `nocto` option may improve performance for read-only mounts, but should be used only if the data on the server changes only occasionally. We have noticed this behavior in an issue about [refs not found after a push](https://gitlab.com/gitlab-org/gitlab/-/issues/326066), where newly added loose refs can be seen as missing on a different client with a local dentry cache, as diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 7e6dc51d27c..ddeaf0280eb 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -76,7 +76,7 @@ because it does not require a shared folder. Consolidated object storage configuration can't be used for backups or Mattermost. See the [full table for a complete list](#storage-specific-configuration). -However, backups can be configured with [server side encryption](../raketasks/backup_restore.md#s3-encrypted-buckets) separately. +However, backups can be configured with [server side encryption](../raketasks/backup_gitlab.md#s3-encrypted-buckets) separately. Enabling consolidated object storage enables object storage for all object types. If not all buckets are specified, `sudo gitlab-ctl reconfigure` may fail with the error like: @@ -528,7 +528,7 @@ supported by consolidated configuration form, refer to the following guides: | Object storage type | Supported by consolidated configuration? | |---------------------|------------------------------------------| -| [Backups](../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | +| [Backups](../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | **{dotted-circle}** No | | [Job artifacts](job_artifacts.md#using-object-storage) including archived job logs | **{check-circle}** Yes | | [LFS objects](lfs/index.md#storing-lfs-objects-in-remote-object-storage) | **{check-circle}** Yes | | [Uploads](uploads.md#using-object-storage) | **{check-circle}** Yes | @@ -542,7 +542,7 @@ supported by consolidated configuration form, refer to the following guides: | [Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | WARNING: -The use of [encrypted S3 buckets](#encrypted-s3-buckets) with non-consolidated configuration is not supported. +The use of [encrypted S3 buckets](#encrypted-s3-buckets) with non-consolidated configuration is not supported. You may start getting [ETag mismatch errors](#etag-mismatch) if you use it. ### Other alternatives to file system storage @@ -587,7 +587,7 @@ Helm-based installs require separate buckets to ### S3 API compatibility issues -Not all S3 providers [are fully compatible](../raketasks/backup_restore.md#other-s3-providers) +Not all S3 providers [are fully compatible](../raketasks/backup_gitlab.md#other-s3-providers) with the Fog library that GitLab uses. Symptoms include an error in `production.log`: ```plaintext diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index cffe682a80b..373017eefa7 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab allows you to start multiple Sidekiq processes. These processes can be used to consume a dedicated set of queues. This can be used to ensure certain queues always have dedicated -workers, no matter the number of jobs that need to be processed. +workers, no matter the number of jobs to be processed. NOTE: The information in this page applies only to Omnibus GitLab. @@ -325,17 +325,16 @@ you'd use the following: The `sidekiq-cluster` command does not terminate once it has started the desired amount of Sidekiq processes. Instead, the process continues running and -forward any signals to the child processes. This makes it easy to stop all -Sidekiq processes as you simply send a signal to the `sidekiq-cluster` process, +forwards any signals to the child processes. This allows you to stop all +Sidekiq processes as you send a signal to the `sidekiq-cluster` process, instead of having to send it to the individual processes. If the `sidekiq-cluster` process crashes or receives a `SIGKILL`, the child processes terminate themselves after a few seconds. This ensures you don't end up with zombie Sidekiq processes. -All of this makes monitoring the processes fairly easy. Simply hook up -`sidekiq-cluster` to your supervisor of choice (for example, runit) and you're good to -go. +This allows you to monitor the processes by hooking up +`sidekiq-cluster` to your supervisor of choice (for example, runit). If a child process died the `sidekiq-cluster` command signals all remaining process to terminate, then terminate itself. This removes the need for diff --git a/doc/administration/operations/extra_sidekiq_routing.md b/doc/administration/operations/extra_sidekiq_routing.md index 2a14fa1d312..a6ad3e62bb7 100644 --- a/doc/administration/operations/extra_sidekiq_routing.md +++ b/doc/administration/operations/extra_sidekiq_routing.md @@ -75,7 +75,7 @@ workers. ## Worker matching query -GitLab provides a simple query syntax to match a worker based on its +GitLab provides a query syntax to match a worker based on its attributes. This query syntax is employed by both [Queue routing rules](#queue-routing-rules) and [Queue selector](extra_sidekiq_processes.md#queue-selector). A query includes two @@ -102,12 +102,8 @@ based on a subset of worker attributes: quickly. Can be `high`, `low`, or `throttled`. For example, the `authorized_projects` queue is used to refresh user permissions, and is `high` urgency. -- `worker_name` - the worker name. The other attributes are typically more useful as - they are more general, but this is available in case a particular worker needs - to be selected. -- `name` - the queue name generated from the worker name. The other attributes - are typically more useful as they are more general, but this is available in - case a particular queue needs to be selected. Because this is generated from +- `worker_name` - the worker name. Use this attribute to select a specific worker. +- `name` - the queue name generated from the worker name. Use this attribute to select a specific queue. Because this is generated from the worker name, it does not change based on the result of other routing rules. - `resource_boundary` - if the queue is bound by `cpu`, `memory`, or @@ -137,7 +133,7 @@ GitLab Enterprise Edition](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee to lowest precedence: - `|` - the logical `OR` operator. For example, `query_a|query_b` (where `query_a` - and `query_b` are queries made up of the other operators here) will include + and `query_b` are queries made up of the other operators here) includes queues that match either query. - `&` - the logical `AND` operator. For example, `query_a&query_b` (where `query_a` and `query_b` are queries made up of the other operators here) will @@ -161,7 +157,7 @@ entire queue group selects all queues. ### Migration -After the Sidekiq routing rules are changed, administrators need to take care +After the Sidekiq routing rules are changed, administrators must take care with the migration to avoid losing jobs entirely, especially in a system with long queues of jobs. The migration can be done by following the migration steps mentioned in [Sidekiq job @@ -181,7 +177,7 @@ sidekiq['routing_rules'] = [ ] ``` -These queues will also need to be included in at least one [Sidekiq +These queues must also be included in at least one [Sidekiq queue group](extra_sidekiq_processes.md#start-multiple-processes). The following table shows the workers that should have their own queue: diff --git a/doc/administration/operations/filesystem_benchmarking.md b/doc/administration/operations/filesystem_benchmarking.md index 9403634d905..a5c4795efea 100644 --- a/doc/administration/operations/filesystem_benchmarking.md +++ b/doc/administration/operations/filesystem_benchmarking.md @@ -36,7 +36,7 @@ fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --bs ``` This creates a 4GB file in `/path/to/git-data/testfile`. It performs -4KB reads and writes using a 75%/25% split within the file, with 64 +4KB reads and writes using a 75%/25% split in the file, with 64 operations running at a time. Be sure to delete the file after the test completes. @@ -72,7 +72,7 @@ operations per second. ### Simple benchmarking NOTE: -This test is naive but may be useful if `fio` is not +This test is naive but can be used if `fio` is not available on the system. It's possible to receive good results on this test but still have poor performance due to read speed and various other factors. @@ -83,7 +83,7 @@ executed, and then reads the same 1,000 files. 1. Change into the root of the appropriate [repository storage path](../repository_storage_paths.md). -1. Create a temporary directory for the test so it's easy to remove the files later: +1. Create a temporary directory for the test so it can be removed later: ```shell mkdir test; cd test diff --git a/doc/administration/operations/index.md b/doc/administration/operations/index.md index fe531407eb2..41a9a0e6d67 100644 --- a/doc/administration/operations/index.md +++ b/doc/administration/operations/index.md @@ -15,7 +15,7 @@ Keep your GitLab instance up and running smoothly. by GitLab to another file system or another server. - [Sidekiq MemoryKiller](sidekiq_memory_killer.md): Configure Sidekiq MemoryKiller to restart Sidekiq. -- [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that need to be processed. **(FREE SELF)** +- [Multiple Sidekiq processes](extra_sidekiq_processes.md): Configure multiple Sidekiq processes to ensure certain queues always have dedicated workers, no matter the number of jobs that must be processed. **(FREE SELF)** - [Sidekiq routing rules](extra_sidekiq_routing.md): Configure the routing rules to route a job from a worker to a desirable queue. **(FREE SELF)** - [Puma](puma.md): Understand Puma and puma-worker-killer. - Speed up SSH operations by [Authorizing SSH users via a fast, @@ -24,10 +24,10 @@ Keep your GitLab instance up and running smoothly. of SSH certificates](ssh_certificates.md). - [File System Performance Benchmarking](filesystem_benchmarking.md): File system performance can have a big impact on GitLab performance, especially for actions - that read or write Git repositories. This information will help benchmark + that read or write Git repositories. This information helps benchmark file system performance against known good and bad real-world systems. - [The Rails Console](rails_console.md): Provides a way to interact with your GitLab instance from the command line. Used for troubleshooting a problem or retrieving some data that can only be done through direct access to GitLab. - [ChatOps Scripts](https://gitlab.com/gitlab-com/chatops): The GitLab.com Infrastructure team uses this repository to house common ChatOps scripts they use to troubleshoot and maintain the production instance of GitLab.com. - These scripts are likely useful to administrators of GitLab instances of all sizes. + These scripts can be used by administrators of GitLab instances of all sizes. diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 1459b707e5c..4228b792fdc 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Moving repositories managed by GitLab **(FREE SELF)** @@ -189,9 +188,9 @@ should be used. Git repositories are accessed, managed, and stored on GitLab ser can result from directly accessing and copying Gitaly's files using tools like `rsync`. - From GitLab 13.3, backup performance can be improved by - [processing multiple repositories concurrently](../../raketasks/backup_restore.md#back-up-git-repositories-concurrently). + [processing multiple repositories concurrently](../../raketasks/backup_gitlab.md#back-up-git-repositories-concurrently). - Backups can be created of just the repositories using the - [skip feature](../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup). + [skip feature](../../raketasks/backup_gitlab.md#excluding-specific-directories-from-the-backup). No other method works for Gitaly Cluster targets. @@ -385,6 +384,6 @@ See the following for information on troubleshooting repository moves. ### Repository move fails for archived projects Because of a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/363670), -[archived projects](../../user/project/settings/index.md#advanced-settings) fail to move even though the data is cloned +[archived projects](../../user/project/settings/index.md#advanced-project-settings) fail to move even though the data is cloned by Gitaly. Make sure archived projects are -[unarchived](../../user/project/settings/index.md#unarchiving-a-project) before initiating a move. +[unarchived](../../user/project/settings/index.md#unarchive-a-project) before initiating a move. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 0025ca109e5..bc95ee626ce 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -111,7 +111,7 @@ To change the worker timeout to 600 seconds: WARNING: This is an experimental [Alpha feature](../../policy/alpha-beta-support.md#alpha-features) and subject to change without notice. The feature is not ready for production use. If you want to use this feature, we recommend testing -with non-production data first. See the [known issues](#puma-single-mode-known-issues) +outside of production first. See the [known issues](#puma-single-mode-known-issues) for additional details. In a memory-constrained environment with less than 4GB of RAM available, consider disabling Puma @@ -137,6 +137,10 @@ For details on Puma worker and thread settings, see the [Puma requirements](../. The downside of running Puma in this configuration is the reduced throughput, which can be considered a fair tradeoff in a memory-constrained environment. +Remember to have sufficient swap available to avoid out of memory (OOM) +conditions. View the [Memory requirements](../../install/requirements.md#memory) +for details. + ### Puma single mode known issues When running Puma in single mode, some features are not supported: @@ -279,6 +283,124 @@ To switch from Unicorn to Puma: 1. Optional. For multi-node deployments, configure the load balancer to use the [readiness check](../load_balancer.md#readiness-check). +## Troubleshooting Puma + +### 502 Gateway Timeout after Puma spins at 100% CPU + +This error occurs when the Web server times out (default: 60 s) after not +hearing back from the Puma worker. If the CPU spins to 100% while this is in +progress, there may be something taking longer than it should. + +To fix this issue, we first need to figure out what is happening. The +following tips are only recommended if you do not mind users being affected by +downtime. Otherwise, skip to the next section. + +1. Load the problematic URL +1. Run `sudo gdb -p <PID>` to attach to the Puma process. +1. In the GDB window, type: + + ```plaintext + call (void) rb_backtrace() + ``` + +1. This forces the process to generate a Ruby backtrace. Check + `/var/log/gitlab/puma/puma_stderr.log` for the backtrace. For example, you may see: + + ```plaintext + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `loop' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:36:in `block (2 levels) in start' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:44:in `sample' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `sample_objects' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each_with_object' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `block in sample_objects' + from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `name' + ``` + +1. To see the current threads, run: + + ```plaintext + thread apply all bt + ``` + +1. Once you're done debugging with `gdb`, be sure to detach from the process and exit: + + ```plaintext + detach + exit + ``` + +GDB reports an error if the Puma process terminates before you can run these commands. +To buy more time, you can always raise the +Puma worker timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and +increase it from 60 seconds to 600: + +```ruby +gitlab_rails['env'] = { + 'GITLAB_RAILS_RACK_TIMEOUT' => 600 +} +``` + +For source installations, set the environment variable. +Refer to [Puma Worker timeout](../operations/puma.md#change-the-worker-timeout). + +[Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. + +#### Troubleshooting without affecting other users + +The previous section attached to a running Puma process, which may have +undesirable effects on users trying to access GitLab during this time. If you +are concerned about affecting others during a production system, you can run a +separate Rails process to debug the issue: + +1. Log in to your GitLab account. +1. Copy the URL that is causing problems (for example, `https://gitlab.com/ABC`). +1. Create a Personal Access Token for your user (User Settings -> Access Tokens). +1. Bring up the [GitLab Rails console.](../operations/rails_console.md#starting-a-rails-console-session) +1. At the Rails console, run: + + ```ruby + app.get '<URL FROM STEP 2>/?private_token=<TOKEN FROM STEP 3>' + ``` + + For example: + + ```ruby + app.get 'https://gitlab.com/gitlab-org/gitlab-foss/-/issues/1?private_token=123456' + ``` + +1. In a new window, run `top`. It should show this Ruby process using 100% CPU. Write down the PID. +1. Follow step 2 from the previous section on using GDB. + +### GitLab: API is not accessible + +This often occurs when GitLab Shell attempts to request authorization via the +[internal API](../../development/internal_api/index.md) (for example, `http://localhost:8080/api/v4/internal/allowed`), and +something in the check fails. There are many reasons why this may happen: + +1. Timeout connecting to a database (for example, PostgreSQL or Redis) +1. Error in Git hooks or push rules +1. Error accessing the repository (for example, stale NFS handles) + +To diagnose this problem, try to reproduce the problem and then see if there +is a Puma worker that is spinning via `top`. Try to use the `gdb` +techniques above. In addition, using `strace` may help isolate issues: + +```shell +strace -ttTfyyy -s 1024 -p <PID of puma worker> -o /tmp/puma.txt +``` + +If you cannot isolate which Puma worker is the issue, try to run `strace` +on all the Puma workers to see where the +[`/internal/allowed`](../../development/internal_api/index.md) endpoint gets stuck: + +```shell +ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/puma.txt +``` + +The output in `/tmp/puma.txt` may help diagnose the root cause. + ## Related topics - [Use a dedicated metrics server to export web metrics](../monitoring/prometheus/puma_exporter.md) diff --git a/doc/administration/operations/rails_console.md b/doc/administration/operations/rails_console.md index df039ee734f..660a99faaf8 100644 --- a/doc/administration/operations/rails_console.md +++ b/doc/administration/operations/rails_console.md @@ -6,8 +6,10 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Rails console **(FREE SELF)** +At the heart of GitLab is a web application [built using the Ruby on Rails +framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). The [Rails console](https://guides.rubyonrails.org/command_line.html#rails-console). -provides a way to interact with your GitLab instance from the command line. +provides a way to interact with your GitLab instance from the command line, and also grants access to the amazing tools built right into Rails. WARNING: The Rails console interacts directly with GitLab. In many cases, @@ -17,7 +19,9 @@ with no consequences, you are strongly advised to do so in a test environment. The Rails console is for GitLab system administrators who are troubleshooting a problem or need to retrieve some data that can only be done through direct -access of the GitLab application. +access of the GitLab application. Basic knowledge of Ruby is needed (try [this +30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction). +Rails experience is useful but not required. ## Starting a Rails console session @@ -35,10 +39,39 @@ sudo -u git -H bundle exec rails console -e production **For Kubernetes deployments** -The console is in the toolbox pod. Refer to our [Kubernetes cheat sheet](../troubleshooting/kubernetes_cheat_sheet.md#gitlab-specific-kubernetes-information) for details. +The console is in the toolbox pod. Refer to our [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html#gitlab-specific-kubernetes-information) for details. To exit the console, type: `quit`. +## Enable Active Record logging + +You can enable output of Active Record debug logging in the Rails console +session by running: + +```ruby +ActiveRecord::Base.logger = Logger.new($stdout) +``` + +This shows information about database queries triggered by any Ruby code +you may run in the console. To turn off logging again, run: + +```ruby +ActiveRecord::Base.logger = nil +``` + +## Disable database statement timeout + +You can disable the PostgreSQL statement timeout for the current Rails console +session by running: + +```ruby +ActiveRecord::Base.connection.execute('SET statement_timeout TO 0') +``` + +This change only affects the current Rails console session and is +not persisted in the GitLab production environment or in the next Rails +console session. + ## Output Rails console session history Enter the following command on the rails console to display @@ -168,3 +201,435 @@ sudo chown -R git:git /scripts sudo chmod 700 /scripts sudo gitlab-rails runner /scripts/helloworld.rb ``` + +## Active Record objects + +### Looking up database-persisted objects + +Under the hood, Rails uses [Active Record](https://guides.rubyonrails.org/active_record_basics.html), +an object-relational mapping system, to read, write, and map application objects +to the PostgreSQL database. These mappings are handled by Active Record models, +which are Ruby classes defined in a Rails app. For GitLab, the model classes +can be found at `/opt/gitlab/embedded/service/gitlab-rails/app/models`. + +Let's enable debug logging for Active Record so we can see the underlying +database queries made: + +```ruby +ActiveRecord::Base.logger = Logger.new($stdout) +``` + +Now, let's try retrieving a user from the database: + +```ruby +user = User.find(1) +``` + +Which would return: + +```ruby +D, [2020-03-05T16:46:25.571238 #910] DEBUG -- : User Load (1.8ms) SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1 +=> #<User id:1 @root> +``` + +We can see that we've queried the `users` table in the database for a row whose +`id` column has the value `1`, and Active Record has translated that database +record into a Ruby object that we can interact with. Try some of the following: + +- `user.username` +- `user.created_at` +- `user.admin` + +By convention, column names are directly translated into Ruby object attributes, +so you should be able to do `user.<column_name>` to view the attribute's value. + +Also by convention, Active Record class names (singular and in camel case) map +directly onto table names (plural and in snake case) and vice versa. For example, +the `users` table maps to the `User` class, while the `application_settings` +table maps to the `ApplicationSetting` class. + +You can find a list of tables and column names in the Rails database schema, +available at `/opt/gitlab/embedded/service/gitlab-rails/db/schema.rb`. + +You can also look up an object from the database by attribute name: + +```ruby +user = User.find_by(username: 'root') +``` + +Which would return: + +```ruby +D, [2020-03-05T17:03:24.696493 #910] DEBUG -- : User Load (2.1ms) SELECT "users".* FROM "users" WHERE "users"."username" = 'root' LIMIT 1 +=> #<User id:1 @root> +``` + +Give the following a try: + +- `User.find_by(email: 'admin@example.com')` +- `User.where.not(admin: true)` +- `User.where('created_at < ?', 7.days.ago)` + +Did you notice that the last two commands returned an `ActiveRecord::Relation` +object that appeared to contain multiple `User` objects? + +Up to now, we've been using `.find` or `.find_by`, which are designed to return +only a single object (notice the `LIMIT 1` in the generated SQL query?). +`.where` is used when it is desirable to get a collection of objects. + +Let's get a collection of non-administrator users and see what we can do with it: + +```ruby +users = User.where.not(admin: true) +``` + +Which would return: + +```ruby +D, [2020-03-05T17:11:16.845387 #910] DEBUG -- : User Load (2.8ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE LIMIT 11 +=> #<ActiveRecord::Relation [#<User id:3 @support-bot>, #<User id:7 @alert-bot>, #<User id:5 @carrie>, #<User id:4 @bernice>, #<User id:2 @anne>]> +``` + +Now, try the following: + +- `users.count` +- `users.order(created_at: :desc)` +- `users.where(username: 'support-bot')` + +In the last command, we see that we can chain `.where` statements to generate +more complex queries. Notice also that while the collection returned contains +only a single object, we cannot directly interact with it: + +```ruby +users.where(username: 'support-bot').username +``` + +Which would return: + +```ruby +Traceback (most recent call last): + 1: from (irb):37 +D, [2020-03-05T17:18:25.637607 #910] DEBUG -- : User Load (1.6ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE AND "users"."username" = 'support-bot' LIMIT 11 +NoMethodError (undefined method `username' for #<ActiveRecord::Relation [#<User id:3 @support-bot>]>) +Did you mean? by_username +``` + +Let's retrieve the single object from the collection by using the `.first` +method to get the first item in the collection: + +```ruby +users.where(username: 'support-bot').first.username +``` + +We now get the result we wanted: + +```ruby +D, [2020-03-05T17:18:30.406047 #910] DEBUG -- : User Load (2.6ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE AND "users"."username" = 'support-bot' ORDER BY "users"."id" ASC LIMIT 1 +=> "support-bot" +``` + +For more on different ways to retrieve data from the database using Active +Record, please see the [Active Record Query Interface documentation](https://guides.rubyonrails.org/active_record_querying.html). + +### Modifying Active Record objects + +In the previous section, we learned about retrieving database records using +Active Record. Now, let's learn how to write changes to the database. + +First, let's retrieve the `root` user: + +```ruby +user = User.find_by(username: 'root') +``` + +Next, let's try updating the user's password: + +```ruby +user.password = 'password' +user.save +``` + +Which would return: + +```ruby +Enqueued ActionMailer::MailDeliveryJob (Job ID: 05915c4e-c849-4e14-80bb-696d5ae22065) to Sidekiq(mailers) with arguments: "DeviseMailer", "password_change", "deliver_now", #<GlobalID:0x00007f42d8ccebe8 @uri=#<URI::GID gid://gitlab/User/1>> +=> true +``` + +Here, we see that the `.save` command returned `true`, indicating that the +password change was successfully saved to the database. + +We also see that the save operation triggered some other action -- in this case +a background job to deliver an email notification. This is an example of an +[Active Record callback](https://guides.rubyonrails.org/active_record_callbacks.html) +-- code which is designated to run in response to events in the Active Record +object life cycle. This is also why using the Rails console is preferred when +direct changes to data is necessary as changes made via direct database queries +do not trigger these callbacks. + +It's also possible to update attributes in a single line: + +```ruby +user.update(password: 'password') +``` + +Or update multiple attributes at once: + +```ruby +user.update(password: 'password', email: 'hunter2@example.com') +``` + +Now, let's try something different: + +```ruby +# Retrieve the object again so we get its latest state +user = User.find_by(username: 'root') +user.password = 'password' +user.password_confirmation = 'hunter2' +user.save +``` + +This returns `false`, indicating that the changes we made were not saved to the +database. You can probably guess why, but let's find out for sure: + +```ruby +user.save! +``` + +This should return: + +```ruby +Traceback (most recent call last): + 1: from (irb):64 +ActiveRecord::RecordInvalid (Validation failed: Password confirmation doesn't match Password) +``` + +Aha! We've tripped an [Active Record Validation](https://guides.rubyonrails.org/active_record_validations.html). +Validations are business logic put in place at the application-level to prevent +unwanted data from being saved to the database and in most cases come with +helpful messages letting you know how to fix the problem inputs. + +We can also add the bang (Ruby speak for `!`) to `.update`: + +```ruby +user.update!(password: 'password', password_confirmation: 'hunter2') +``` + +In Ruby, method names ending with `!` are commonly known as "bang methods". By +convention, the bang indicates that the method directly modifies the object it +is acting on, as opposed to returning the transformed result and leaving the +underlying object untouched. For Active Record methods that write to the +database, bang methods also serve an additional function: they raise an +explicit exception whenever an error occurs, instead of just returning `false`. + +We can also skip validations entirely: + +```ruby +# Retrieve the object again so we get its latest state +user = User.find_by(username: 'root') +user.password = 'password' +user.password_confirmation = 'hunter2' +user.save!(validate: false) +``` + +This is not recommended, as validations are usually put in place to ensure the +integrity and consistency of user-provided data. + +A validation error prevents the entire object from being saved to +the database. You can see a little of this in the section below. If you're getting +a mysterious red banner in the GitLab UI when submitting a form, this can often +be the fastest way to get to the root of the problem. + +### Interacting with Active Record objects + +At the end of the day, Active Record objects are just normal Ruby objects. As +such, we can define methods on them which perform arbitrary actions. + +For example, GitLab developers have added some methods which help with +two-factor authentication: + +```ruby +def disable_two_factor! + transaction do + update( + otp_required_for_login: false, + encrypted_otp_secret: nil, + encrypted_otp_secret_iv: nil, + encrypted_otp_secret_salt: nil, + otp_grace_period_started_at: nil, + otp_backup_codes: nil + ) + self.u2f_registrations.destroy_all # rubocop: disable DestroyAll + end +end + +def two_factor_enabled? + two_factor_otp_enabled? || two_factor_u2f_enabled? +end +``` + +(See: `/opt/gitlab/embedded/service/gitlab-rails/app/models/user.rb`) + +We can then use these methods on any user object: + +```ruby +user = User.find_by(username: 'root') +user.two_factor_enabled? +user.disable_two_factor! +``` + +Some methods are defined by gems, or Ruby software packages, which GitLab uses. +For example, the [StateMachines](https://github.com/state-machines/state_machines-activerecord) +gem which GitLab uses to manage user state: + +```ruby +state_machine :state, initial: :active do + event :block do + + ... + + event :activate do + + ... + +end +``` + +Give it a try: + +```ruby +user = User.find_by(username: 'root') +user.state +user.block +user.state +user.activate +user.state +``` + +Earlier, we mentioned that a validation error prevents the entire object +from being saved to the database. Let's see how this can have unexpected +interactions: + +```ruby +user.password = 'password' +user.password_confirmation = 'hunter2' +user.block +``` + +We get `false` returned! Let's find out what happened by adding a bang as we did +earlier: + +```ruby +user.block! +``` + +Which would return: + +```ruby +Traceback (most recent call last): + 1: from (irb):87 +StateMachines::InvalidTransition (Cannot transition state via :block from :active (Reason(s): Password confirmation doesn't match Password)) +``` + +We see that a validation error from what feels like a completely separate +attribute comes back to haunt us when we try to update the user in any way. + +In practical terms, we sometimes see this happen with GitLab administration settings -- +validations are sometimes added or changed in a GitLab update, resulting in +previously saved settings now failing validation. Because you can only update +a subset of settings at once through the UI, in this case the only way to get +back to a good state is direct manipulation via Rails console. + +### Commonly used Active Record models and how to look up objects + +**Get a user by primary email address or username:** + +```ruby +User.find_by(email: 'admin@example.com') +User.find_by(username: 'root') +``` + +**Get a user by primary OR secondary email address:** + +```ruby +User.find_by_any_email('user@example.com') +``` + +The `find_by_any_email` method is a custom method added by GitLab developers rather +than a Rails-provided default method. + +**Get a collection of administrator users:** + +```ruby +User.admins +``` + +`admins` is a [scope convenience method](https://guides.rubyonrails.org/active_record_querying.html#scopes) +which does `where(admin: true)` under the hood. + +**Get a project by its path:** + +```ruby +Project.find_by_full_path('group/subgroup/project') +``` + +`find_by_full_path` is a custom method added by GitLab developers rather +than a Rails-provided default method. + +**Get a project's issue or merge request by its numeric ID:** + +```ruby +project = Project.find_by_full_path('group/subgroup/project') +project.issues.find_by(iid: 42) +project.merge_requests.find_by(iid: 42) +``` + +`iid` means "internal ID" and is how we keep issue and merge request IDs +scoped to each GitLab project. + +**Get a group by its path:** + +```ruby +Group.find_by_full_path('group/subgroup') +``` + +**Get a group's related groups:** + +```ruby +group = Group.find_by_full_path('group/subgroup') + +# Get a group's parent group +group.parent + +# Get a group's child groups +group.children +``` + +**Get a group's projects:** + +```ruby +group = Group.find_by_full_path('group/subgroup') + +# Get group's immediate child projects +group.projects + +# Get group's child projects, including those in subgroups +group.all_projects +``` + +**Get CI pipeline or builds:** + +```ruby +Ci::Pipeline.find(4151) +Ci::Build.find(66124) +``` + +The pipeline and job ID numbers increment globally across your GitLab +instance, so there's no requirement to use an internal ID attribute to look them up, +unlike with issues or merge requests. + +**Get the current application settings object:** + +```ruby +ApplicationSetting.current +``` diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 3141312612e..a777f5484fd 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -96,7 +96,7 @@ Match User git AuthorizedPrincipalsCommand /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell-authorized-principals-check %i sshUsers ``` -This command will emit output that looks something like: +This command emits output that looks something like: ```shell command="/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell username-{KEY_ID}",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty {PRINCIPAL} diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md index 420690866e5..422339c014c 100644 --- a/doc/administration/package_information/deprecation_policy.md +++ b/doc/administration/package_information/deprecation_policy.md @@ -41,7 +41,7 @@ We can differentiate two different types of configuration: installation usable (like a change in default project/group settings, or miscommunication with other components) -We also need to differentiate deprecation and removal procedure. +We must also differentiate deprecation and removal procedure. #### Deprecating configuration @@ -78,7 +78,7 @@ The final comment in the issue **has to have**: 1. Text snippet for the release blog post section 1. Documentation MR ( or snippet ) for introducing the change -1. Draft MR removing the configuration OR details on what needs to be done. See [Adding deprecation messages](https://docs.gitlab.com/omnibus/development/adding-deprecation-messages.html) for more on this +1. Draft MR removing the configuration or details on what must be done. See [Adding deprecation messages](https://docs.gitlab.com/omnibus/development/adding-deprecation-messages.html) for more on this ## Example diff --git a/doc/administration/package_information/postgresql_versions.md b/doc/administration/package_information/postgresql_versions.md index 8aab4121a0b..afdc63c4851 100644 --- a/doc/administration/package_information/postgresql_versions.md +++ b/doc/administration/package_information/postgresql_versions.md @@ -29,18 +29,18 @@ Read more about update policies and warnings in the PostgreSQL | GitLab version | PostgreSQL versions | Default version for fresh installs | Default version for upgrades | Notes | | -------------- | --------------------- | ---------------------------------- | ---------------------------- | ----- | -| 15.0 | 12.10, 13.6 | 13.6 | 12.10 | For upgrades, users can manually upgrade to 13.6 following the [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | +| 15.0 | 12.10, 13.6 | 13.6 | 12.10 | For upgrades, users can manually upgrade to 13.6 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-150-and-later). | | 14.1 | 12.7, 13.3 | 12.7 | 12.7 | PostgreSQL 13 available for fresh installations if not using [Geo](../geo/index.md#requirements-for-running-geo) or [Patroni](../postgresql/index.md#postgresql-replication-and-failover-with-omnibus-gitlab). -| 14.0 | 12.7 | 12.7 | 12.7 | HA installations with repmgr are no longer supported and will be prevented from upgrading to Omnibus GitLab 14.0 | +| 14.0 | 12.7 | 12.7 | 12.7 | HA installations with repmgr are no longer supported and are prevented from upgrading to Omnibus GitLab 14.0 | | 13.8 | 11.9, 12.4 | 12.4 | 12.4 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or HA cluster.). | -| 13.7 | 11.9, 12.4 | 12.4 | 11.9 | For upgrades users can manually upgrade to 12.4 following the [upgrade docs](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-133-and-later). | +| 13.7 | 11.9, 12.4 | 12.4 | 11.9 | For upgrades users can manually upgrade to 12.4 following the [upgrade documentation](https://docs.gitlab.com/omnibus/settings/database.html#gitlab-133-and-later). | | 13.4 | 11.9, 12.4 | 11.9 | 11.9 | Package upgrades aborted if users not running PostgreSQL 11 already | | 13.3 | 11.7, 12.3 | 11.7 | 11.7 | Package upgrades aborted if users not running PostgreSQL 11 already | | 13.0 | 11.7 | 11.7 | 11.7 | Package upgrades aborted if users not running PostgreSQL 11 already | | 12.10 | 9.6.17, 10.12, and 11.7 | 11.7 | 11.7 | Package upgrades automatically performed PostgreSQL upgrade for nodes that are not part of a Geo or repmgr cluster. | -| 12.8 | 9.6.17, 10.12, and 11.7 | 10.12 | 10.12 | Users can manually upgrade to 11.7 following the upgrade docs. | +| 12.8 | 9.6.17, 10.12, and 11.7 | 10.12 | 10.12 | Users can manually upgrade to 11.7 following the upgrade documentation. | | 12.0 | 9.6.11 and 10.7 | 10.7 | 10.7 | Package upgrades automatically performed PostgreSQL upgrade. | -| 11.11 | 9.6.11 and 10.7 | 9.6.11 | 9.6.11 | Users can manually upgrade to 10.7 following the upgrade docs. | +| 11.11 | 9.6.11 and 10.7 | 9.6.11 | 9.6.11 | Users can manually upgrade to 10.7 following the upgrade documentation. | | 10.0 | 9.6.3 | 9.6.3 | 9.6.3 | Package upgrades aborted if users still on 9.2. | | 9.0 | 9.2.18 and 9.6.1 | 9.6.1 | 9.6.1 | Package upgrades automatically performed PostgreSQL upgrade. | -| 8.14 | 9.2.18 and 9.6.1 | 9.2.18 | 9.2.18 | Users can manually upgrade to 9.6 following the upgrade docs. | +| 8.14 | 9.2.18 and 9.6.1 | 9.2.18 | 9.2.18 | Users can manually upgrade to 9.6 following the upgrade documentation. | diff --git a/doc/administration/package_information/supported_os.md b/doc/administration/package_information/supported_os.md index f2838cd779b..4f31981a05c 100644 --- a/doc/administration/package_information/supported_os.md +++ b/doc/administration/package_information/supported_os.md @@ -15,28 +15,27 @@ page](https://about.gitlab.com/install/). The following lists the currently supported OSs and their possible EOL dates. -| OS Version | First supported GitLab version | Arch | Install Docs | OS EOL | Details | +| OS Version | First supported GitLab version | Arch | Install Documentation | OS EOL | Details | | ------------------------------------------------------------ | ------------------------------ | --------------- | :----------------------------------------------------------: | ---------- | ------------------------------------------------------------ | -| AlmaLinux 8 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [AlmaLinux Install Docs](https://about.gitlab.com/install/#almalinux-8) | 2029 | <https://almalinux.org/> | -| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://wiki.centos.org/About/Product> | -| Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | [Debian Install Docs](https://about.gitlab.com/install/#debian) | 2022 | <https://wiki.debian.org/LTS> | -| Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Docs](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | -| Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Docs](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | -| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [OpenSUSE Install Docs](https://about.gitlab.com/install/#opensuse-leap-15-3) | Nov 2022 | <https://en.opensuse.org/Lifetime> | -| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | May 2024 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | -| SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Docs](https://about.gitlab.com/install/#opensuse-leap-15-3) | Oct 2027 | <https://www.suse.com/lifecycle/> | - -| Oracle Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | Jul 2024 | <https://www.oracle.com/a/ocom/docs/elsp-lifetime-069338.pdf> | -| Scientific Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Docs](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://scientificlinux.org/downloads/sl-versions/sl7/> | -| Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | [Ubuntu Install Docs](https://about.gitlab.com/install/#ubuntu) | April 2023 | <https://wiki.ubuntu.com/Releases> | -| Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | [Ubuntu Install Docs](https://about.gitlab.com/install/#ubuntu) | April 2025 | <https://wiki.ubuntu.com/Releases> | -| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | [Amazon Linux 2 Instal Docsl](https://about.gitlab.com/install/#amazonlinux-2) | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs/> | -| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | [Raspberry Pi Install Docs](https://about.gitlab.com/install/#raspberry-pi-os) | 2024 | [Raspberry Pi Details](https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/) | +| AlmaLinux 8 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [AlmaLinux Install Documentation](https://about.gitlab.com/install/#almalinux-8) | 2029 | <https://almalinux.org/> | +| CentOS 7 | GitLab CE / GitLab EE 7.10.0 | x86_64 | [CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://wiki.centos.org/About/Product> | +| Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2022 | <https://wiki.debian.org/LTS> | +| Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2024 | <https://wiki.debian.org/LTS> | +| Debian 11 | GitLab CE / GitLab EE 14.6.0 | amd64, arm64 | [Debian Install Documentation](https://about.gitlab.com/install/#debian) | 2026 | <https://wiki.debian.org/LTS> | +| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | [OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap-15-3) | Nov 2022 | <https://en.opensuse.org/Lifetime> | +| RHEL 8 | GitLab CE / GitLab EE 12.8.1 | x86_64, arm64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | May 2024 | [RHEL Details](https://access.redhat.com/support/policy/updates/errata/#Life_Cycle_Dates) | +| SLES 12 | GitLab EE 9.0.0 | x86_64 | [Use OpenSUSE Install Documentation](https://about.gitlab.com/install/#opensuse-leap-15-3) | Oct 2027 | <https://www.suse.com/lifecycle/> | +| Oracle Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | Jul 2024 | <https://www.oracle.com/a/ocom/docs/elsp-lifetime-069338.pdf> | +| Scientific Linux | GitLab CE / GitLab EE 8.14.0 | x86_64 | [Use CentOS Install Documentation](https://about.gitlab.com/install/#centos-7) | June 2024 | <https://scientificlinux.org/downloads/sl-versions/sl7/> | +| Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2023 | <https://wiki.ubuntu.com/Releases> | +| Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | [Ubuntu Install Documentation](https://about.gitlab.com/install/#ubuntu) | April 2025 | <https://wiki.ubuntu.com/Releases> | +| Amazon Linux 2 | GitLab CE / GitLab EE 14.9.0 | amd64, arm64 | [Amazon Linux 2 Install Documentation](https://about.gitlab.com/install/#amazonlinux-2) | June 2023 | <https://aws.amazon.com/amazon-linux-2/faqs/> | +| Raspberry Pi OS (Buster) (formerly known as Raspbian Buster) | GitLab CE 12.2.0 | armhf | [Raspberry Pi Install Documentation](https://about.gitlab.com/install/#raspberry-pi-os) | 2024 | [Raspberry Pi Details](https://www.raspberrypi.com/news/new-old-functionality-with-raspberry-pi-os-legacy/) | NOTE: CentOS 8 was EOL on December 31, 2021. In GitLab 14.5 and later, [CentOS builds work in AlmaLinux](https://gitlab.com/gitlab-org/distribution/team-tasks/-/issues/954#note_730198505). -We will officially support all distributions that are binary compatible with Red Hat Enterprise Linux. +We officially support all distributions that are binary compatible with Red Hat Enterprise Linux. This gives users a path forward for their CentOS 8 builds at its end of life. ## Update GitLab package sources after upgrading the OS @@ -48,7 +47,7 @@ If your package manager reports that no further updates are available, although [new versions have been released](https://about.gitlab.com/releases/categories/releases/), repeat the "Add the GitLab package repository" instructions of the [Linux package install guide](https://about.gitlab.com/install/#content). -Future GitLab upgrades will now be fetched according to your upgraded OS. +Future GitLab upgrades are fetched according to your upgraded OS. ## Packages for ARM64 @@ -65,7 +64,7 @@ running GitLab on ARM. ## OS Versions that are no longer supported GitLab provides omnibus packages for operating systems only until their -EOL (End-Of-Life). After the EOL date of the OS, GitLab will stop releasing +EOL (End-Of-Life). After the EOL date of the OS, GitLab stops releasing official packages. The list of deprecated operating systems and the final GitLab release for them can be found below: diff --git a/doc/administration/packages/container_registry.md b/doc/administration/packages/container_registry.md index 2fa3eea54bd..3bfb3ceca86 100644 --- a/doc/administration/packages/container_registry.md +++ b/doc/administration/packages/container_registry.md @@ -887,7 +887,7 @@ administrators can clean up image tags and [run garbage collection](#container-registry-garbage-collection). To remove image tags by running the cleanup policy, run the following commands in the -[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md): +[GitLab Rails console](../operations/rails_console.md): ```ruby # Numeric ID of the project whose container registry should be cleaned up @@ -927,9 +927,9 @@ these controls should migrate to the GitLab interface. Users who have the [Maintainer role](../../user/permissions.md) for the project can [delete Container Registry tags in bulk](../../api/container_registry.md#delete-registry-repository-tags-in-bulk) -periodically based on their own criteria, however, this alone does not recycle data, +periodically based on their own criteria. However, deleting the tags alone does not recycle data, it only unlinks tags from manifests and image blobs. To recycle the Container -Registry data in the whole GitLab instance, you can use the built-in command +Registry data in the whole GitLab instance, you can use the built-in garbage collection command provided by `gitlab-ctl`. Prerequisites: @@ -938,7 +938,7 @@ Prerequisites: [cloud native chart](https://docs.gitlab.com/charts/charts/registry/#garbage-collection). - You must set the Registry to [read-only mode](#performing-garbage-collection-without-downtime). Running garbage collection causes downtime for the Container Registry. When you run this command - on an instance in an environment where another instances is still writing to the Registry storage, + on an instance in an environment where another instance is still writing to the Registry storage, referenced manifests are removed. ### Understanding the content-addressable layers @@ -1389,7 +1389,7 @@ project or branch name. Special characters can include: - Double hyphen/dash To get around this, you can [change the group path](../../user/group/index.md#change-a-groups-path), -[change the project path](../../user/project/settings/index.md#renaming-a-repository) or change the +[change the project path](../../user/project/settings/index.md#rename-a-repository) or change the branch name. Another option is to create a [push rule](../../user/project/repository/push_rules.md) to prevent this at the instance level. @@ -1452,7 +1452,7 @@ curl "localhost:5001/debug/vars" ### Access old schema v1 Docker images -Support for the [Docker registry v1 API](https://www.docker.com/blog/registry-v1-api-deprecation/), +Support for the Docker registry API V1, including [schema V1 image manifests](https://docs.docker.com/registry/spec/manifest-v2-1/), was: @@ -1738,7 +1738,7 @@ In this case, follow these steps: 1. Try the removal again. If you still can't remove the repository using the common methods, you can use the -[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md) +[GitLab Rails console](../operations/rails_console.md) to remove the project by force: ```ruby diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 1535de0c2c2..b58eeff48cf 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -269,6 +269,9 @@ control over how the Pages daemon runs and serves content in your environment. | `max_uri_length` | The maximum length of URIs accepted by GitLab Pages. Set to 0 for unlimited length. [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/659) in GitLab 14.5. | `metrics_address` | The address to listen on for metrics requests. | | `redirect_http` | Redirect pages from HTTP to HTTPS, true/false. | +| `redirects_max_config_size` | The maximum size of the _redirects file, in bytes (default: 65536). | +| `redirects_max_path_segments` | The maximum number of path segments allowed in _redirects rules URLs (default: 25). | +| `redirects_max_rule_count` | The maximum number of rules allowed in _redirects (default: 1000). | | `sentry_dsn` | The address for sending Sentry crash reporting to. | | `sentry_enabled` | Enable reporting and logging with Sentry, true/false. | | `sentry_environment` | The environment for Sentry crash reporting. | @@ -489,11 +492,6 @@ To do that: 1. Select the **Disable public access to Pages sites** checkbox. 1. Select **Save changes**. -WARNING: -For self-managed installations, all public websites remain private until they are -redeployed. Resolve this issue by -[sourcing domain configuration from the GitLab API](https://gitlab.com/gitlab-org/gitlab/-/issues/218357). - ### Running behind a proxy Like the rest of GitLab, Pages can be used in those environments where external @@ -582,6 +580,19 @@ HTTP Strict Transport Security (HSTS) can be enabled through the `gitlab_pages[' gitlab_pages['headers'] = ['Strict-Transport-Security: max-age=63072000'] ``` +### Pages project redirects limits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/merge_requests/778) in GitLab 15.2. + +GitLab Pages comes with a set of default limits for the [`_redirects` file](../../user/project/pages/redirects.md) +to minimize the impact on performance. You can configure these limits if you'd like to increase or decrease the limits. + +```ruby +gitlab_pages['redirects_max_config_size'] = 131072 +gitlab_pages['redirects_max_path_segments'] = 50 +gitlab_pages['redirects_max_rule_count'] = 2000 +``` + ## Activate verbose logging for daemon Follow the steps below to configure verbose logging of GitLab Pages daemon. @@ -1400,7 +1411,7 @@ this setting needs to be configured on the main GitLab server. If the wildcard DNS [prerequisite](#prerequisites) can't be met, you can still use GitLab Pages in a limited fashion: -1. [Move](../../user/project/settings/index.md#transferring-an-existing-project-into-another-namespace) +1. [Move](../../user/project/settings/index.md#transfer-a-project-to-another-namespace) all projects you need to use Pages with into a single group namespace, for example `pages`. 1. Configure a [DNS entry](#dns-configuration) without the `*.`-wildcard, for example `pages.example.io`. 1. Configure `pages_external_url http://example.io/` in your `gitlab.rb` file. diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 5d693793a92..7036502e377 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -21,7 +21,7 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: 1. If you are using a cloud-managed service, you may need to grant additional roles to your `gitlab` user: - Amazon RDS requires the [`rds_superuser`](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Appendix.PostgreSQL.CommonDBATasks.html#Appendix.PostgreSQL.CommonDBATasks.Roles) role. - - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/howto-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. Azure Database for PostgreSQL - Flexible Server requires [allow-listing extensions before they can be installed](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions). + - Azure Database for PostgreSQL requires the [`azure_pg_admin`](https://docs.microsoft.com/en-us/azure/postgresql/single-server/how-to-create-users#how-to-create-additional-admin-users-in-azure-database-for-postgresql) role. Azure Database for PostgreSQL - Flexible Server requires [allow-listing extensions before they can be installed](https://docs.microsoft.com/en-us/azure/postgresql/flexible-server/concepts-extensions#how-to-use-postgresql-extensions). - Google Cloud SQL requires the [`cloudsqlsuperuser`](https://cloud.google.com/sql/docs/postgres/users#default-users) role. This is for the installation of extensions during installation and upgrades. As an alternative, @@ -48,3 +48,20 @@ If you use a cloud-managed service, or provide your own PostgreSQL instance: ```shell sudo gitlab-ctl reconfigure ``` + +## Troubleshooting + +### Resolve `SSL SYSCALL error: EOF detected` error + +When using an external PostgreSQL instance, you may see an error like: + +```shell +pg_dump: error: Error message from server: SSL SYSCALL error: EOF detected +``` + +To resolve this error, ensure that you are meeting the +[minimum PostgreSQL requirements](../../install/requirements.md#postgresql-requirements). After +upgrading your RDS instance to a suitable version, you should be able to perform a backup without +this error. Refer to issue #64763 +([Segmentation fault citing `LooseForeignKeys::CleanupWorker` causes complete database restart](https://gitlab.com/gitlab-org/gitlab/-/issues/364763)) +for more information. diff --git a/doc/administration/postgresql/moving.md b/doc/administration/postgresql/moving.md new file mode 100644 index 00000000000..23d377dba37 --- /dev/null +++ b/doc/administration/postgresql/moving.md @@ -0,0 +1,65 @@ +--- +stage: Data Stores +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Moving GitLab databases to a different PostgreSQL instance **(FREE SELF)** + +Sometimes it is necessary to move your databases from one PostgreSQL instance to +another. For example, if you are using AWS Aurora and are preparing to +enable Database Load Balancing, you will need to move your databases to +RDS for PostgreSQL. + +To move databases from one instance to another: + +1. Gather the source and destination PostgreSQL endpoint information: + + ```shell + SRC_PGHOST=<source postgresql host> + SRC_PGUSER=<source postgresql user> + + DST_PGHOST=<destination postgresql host> + DST_PGUSER=<destination postgresql user> + ``` + +1. Stop GitLab: + + ```shell + sudo gitlab-ctl stop + ``` + +1. Dump the databases from the source: + + ```shell + /opt/gitlab/embedded/bin/pg_dump -h $SRC_PGHOST -U $SRC_PGUSER -c -C -f gitlabhq_production.sql gitlabhq_production + /opt/gitlab/embedded/bin/pg_dump -h $SRC_PGHOST -U $SRC_PGUSER -c -C -f praefect_production.sql praefect_production + ``` + +1. Restore the databases to the destination (this will overwrite any existing databases with the same names): + + ```shell + /opt/gitlab/embedded/bin/psql -h $DST_PGHOST -U $DST_PGUSER -f praefect_production.sql postgres + /opt/gitlab/embedded/bin/psql -h $DST_PGHOST -U $DST_PGUSER -f gitlabhq_production.sql postgres + ``` + +1. Configure the GitLab application servers with the appropriate connection details + for your destination PostgreSQL instance in your `/etc/gitlab/gitlab.rb` file: + + ```ruby + gitlab_rails['db_host'] = '<destination postgresql host>' + ``` + + For more information on GitLab multi-node setups, refer to the [reference architectures](../reference_architectures/index.md). + +1. Reconfigure for the changes to take effect: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Restart GitLab: + + ```shell + sudo gitlab-ctl start + ``` diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index 8ae2b6497f8..b12edd4b9ad 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -173,7 +173,7 @@ ote_pid | tls Some database changes have to be done directly, and not through PgBouncer. Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer) -and [GitLab upgrades](../../update/zero_downtime.md#use-postgresql-ha). +and [GitLab upgrades](../../update/zero_downtime.md#postgresql). 1. To find the primary node, run the following on a database node: diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index e9b607ad5d4..37471a4f491 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -996,7 +996,7 @@ You can switch an exiting database cluster to use Patroni instead of repmgr with ### Upgrading PostgreSQL major version in a Patroni cluster -As of GitLab 13.3, PostgreSQL 11.7 and 12.3 are both shipped with Omnibus GitLab by default. As of GitLab 13.7, PostgreSQL 12 is the default. If you want to upgrade to PostgreSQL 12 in versions prior to GitLab 13.7, you must ask for it explicitly. +As of GitLab 14.1, PostgreSQL 12.6 and 13.3 are both shipped with Omnibus GitLab by default. As of GitLab 15.0, PostgreSQL 13 is the default. If you want to upgrade to PostgreSQL 13 in versions prior to GitLab 15.0, you must ask for it explicitly. WARNING: The procedure for upgrading PostgreSQL in a Patroni cluster is different than when upgrading using repmgr. @@ -1046,7 +1046,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: 1. Upgrade PostgreSQL on **the leader node** and make sure that the upgrade is completed successfully: ```shell - sudo gitlab-ctl pg-upgrade -V 12 + sudo gitlab-ctl pg-upgrade -V 13 ``` NOTE: @@ -1073,7 +1073,7 @@ Considering these, you should carefully plan your PostgreSQL upgrade: 1. Upgrade PostgreSQL **on replicas** (you can do this in parallel on all of them): ```shell - sudo gitlab-ctl pg-upgrade -V 12 + sudo gitlab-ctl pg-upgrade -V 13 ``` NOTE: diff --git a/doc/administration/raketasks/check.md b/doc/administration/raketasks/check.md index d66ff9afb94..cf569cd81d0 100644 --- a/doc/administration/raketasks/check.md +++ b/doc/administration/raketasks/check.md @@ -211,7 +211,7 @@ secrets file (`gitlab-secrets.json`). Automatic resolution is not yet implemented. If you have values that cannot be decrypted, you can follow steps to reset them, see our -docs on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). +documentation on what to do [when the secrets file is lost](../../raketasks/backup_restore.md#when-the-secrets-file-is-lost). This can take a very long time, depending on the size of your database, as it checks all rows in all tables. @@ -352,23 +352,23 @@ When this scenario is detected, the Rake task displays an error message. For exa ```shell Checking integrity of Job artifacts -- 3..8: Failures: 2 - - Job artifact: 3: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/2021_05_26/5/3/job.log> - - Job artifact: 8: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce/2021_05_26/6/8/job.log> +- 1..15: Failures: 2 + - Job artifact: 9: #<Errno::ENOENT: No such file or directory @ rb_sysopen - /var/opt/gitlab/gitlab-rails/shared/artifacts/4b/22/4b227777d4dd1fc61c6f884f48641d02b4d121d3fd328cb08b5531fcacdabf8a/2022_06_30/8/9/job.log> + - Job artifact: 15: Remote object does not exist Done! ``` -To delete these references to missing local artifacts (`job.log` files): +To delete these references to missing local and/or remote artifacts (`job.log` files): 1. Open the [GitLab Rails Console](../operations/rails_console.md#starting-a-rails-console-session). 1. Run the following Ruby code: ```ruby artifacts_deleted = 0 - ::Ci::JobArtifact.find_each do |artifact| ### Iterate artifacts + ::Ci::JobArtifact.find_each do |artifact| ### Iterate artifacts # next if artifact.file.filename != "job.log" ### Uncomment if only `job.log` files' references are to be processed - next if artifact.file.exists? ### Skip if the file reference is valid + next if artifact.file.file.exists? ### Skip if the file reference is valid artifacts_deleted += 1 puts "#{artifact.id} #{artifact.file.path} is missing." ### Allow verification before destroy # artifact.destroy! ### Uncomment to actually destroy diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index 30c8518c209..0d724bfd4dc 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -11,13 +11,13 @@ A username should be passed as the second argument to the Rake task, which becomes the owner of the project. You can resume an import with the same command. -Bear in mind that the syntax is very specific. Remove any spaces within the argument block and +Bear in mind that the syntax is very specific. Remove any spaces in the argument block and before/after the brackets. Also, some shells (for example, Zsh) can interpret the open/close brackets -(`[]`) separately. You may need to either escape the brackets or use double quotes. +(`[]`) separately. You may want to either escape the brackets or use double quotes. ## Caveats -If the GitHub [rate limit](https://docs.github.com/en/rest/reference/rate-limit) is reached while +If the GitHub [rate limit](https://docs.github.com/en/rest/rate-limit) is reached while importing, the importing process waits (`sleep()`) until it can continue importing. ## Importing multiple projects diff --git a/doc/administration/raketasks/maintenance.md b/doc/administration/raketasks/maintenance.md index f4bd6f255e5..6de47c37957 100644 --- a/doc/administration/raketasks/maintenance.md +++ b/doc/administration/raketasks/maintenance.md @@ -186,7 +186,7 @@ Checking GitLab ... Finished ## Rebuild `authorized_keys` file In some cases it is necessary to rebuild the `authorized_keys` file, -for example, if after an upgrade you receive `Permission denied (publickey)` when pushing [via SSH](../../ssh/index.md) +for example, if after an upgrade you receive `Permission denied (publickey)` when pushing [via SSH](../../user/ssh.md) and find `404 Key Not Found` errors in [the `gitlab-shell.log` file](../logs.md#gitlab-shelllog). To rebuild `authorized_keys`, run: @@ -234,7 +234,7 @@ sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production Sometimes during version upgrades you might end up with some wrong CSS or missing some icons. In that case, try to precompile the assets again. -This only applies to source installations and does NOT apply to +This only applies to source installations and does not apply to Omnibus packages. **Source Installation** diff --git a/doc/administration/raketasks/praefect.md b/doc/administration/raketasks/praefect.md index d2fd4943c68..b7db3f26a60 100644 --- a/doc/administration/raketasks/praefect.md +++ b/doc/administration/raketasks/praefect.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Praefect Rake tasks **(FREE SELF)** diff --git a/doc/administration/raketasks/smtp.md b/doc/administration/raketasks/smtp.md index bc78d54aa1b..49274501809 100644 --- a/doc/administration/raketasks/smtp.md +++ b/doc/administration/raketasks/smtp.md @@ -55,7 +55,7 @@ bundle exec rake gitlab:smtp:secret:edit RAILS_ENV=production EDITOR=vim ### Write raw secret -Write new secret content by providing it on STDIN. +Write new secret content by providing it on `STDIN`. **Omnibus Installation** diff --git a/doc/administration/read_only_gitlab.md b/doc/administration/read_only_gitlab.md index a601ba4cd7a..387a1a75393 100644 --- a/doc/administration/read_only_gitlab.md +++ b/doc/administration/read_only_gitlab.md @@ -63,7 +63,7 @@ sudo gitlab-ctl start puma ## Make the database read-only -If you want to allow users to use the GitLab UI, then you need to ensure that +If you want to allow users to use the GitLab UI, ensure that the database is read-only: 1. Take a [GitLab backup](../raketasks/backup_restore.md) @@ -113,7 +113,7 @@ the database is read-only: sudo gitlab-ctl restart postgresql ``` -When you're ready to revert the read-only state, you need to remove the added +When you're ready to revert the read-only state, remove the added lines in `/etc/gitlab/gitlab.rb`, and reconfigure GitLab and restart PostgreSQL: ```shell @@ -121,5 +121,5 @@ sudo gitlab-ctl reconfigure sudo gitlab-ctl restart postgresql ``` -Once you verify all works as expected, you can remove the `gitlab_read_only` +After you verify all works as expected, remove the `gitlab_read_only` user from the database. diff --git a/doc/administration/redis/index.md b/doc/administration/redis/index.md index 3581b27a4c8..fb5f6cc3a54 100644 --- a/doc/administration/redis/index.md +++ b/doc/administration/redis/index.md @@ -12,7 +12,7 @@ multiple ways to configure Redis. You can choose to install and manage Redis and Sentinel yourself, use a hosted cloud solution, or you can use the ones that come bundled with the Omnibus GitLab -packages so you only need to focus on configuration. Pick the one that suits your needs. +packages so you can only focus on configuration. Pick the one that suits your needs. ## Redis replication and failover using Omnibus GitLab diff --git a/doc/administration/redis/standalone.md b/doc/administration/redis/standalone.md index 6e0e1eb886e..bd5b30efb7b 100644 --- a/doc/administration/redis/standalone.md +++ b/doc/administration/redis/standalone.md @@ -44,7 +44,7 @@ Omnibus GitLab: 1. [Reconfigure Omnibus GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. 1. Note the Redis node's IP address or hostname, port, and - Redis password. These will be necessary when [configuring the GitLab application servers](#set-up-the-gitlab-rails-application-instance). + Redis password. These are necessary when [configuring the GitLab application servers](#set-up-the-gitlab-rails-application-instance). [Advanced configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) are supported and can be added if needed. diff --git a/doc/administration/redis/troubleshooting.md b/doc/administration/redis/troubleshooting.md index 4426adc5d51..ca52fe0a29a 100644 --- a/doc/administration/redis/troubleshooting.md +++ b/doc/administration/redis/troubleshooting.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Troubleshooting Redis **(FREE SELF)** -There are a lot of moving parts that needs to be taken care carefully +There are a lot of moving parts that must be taken care carefully in order for the HA setup to work as expected. Before proceeding with the troubleshooting below, check your firewall rules: @@ -41,7 +41,7 @@ You can check if everything is correct by connecting to each server using /opt/gitlab/embedded/bin/redis-cli -h <redis-host-or-ip> -a '<redis-password>' info replication ``` -When connected to a `Primary` Redis, you will see the number of connected +When connected to a `Primary` Redis, you see the number of connected `replicas`, and a list of each with connection details: ```plaintext @@ -56,7 +56,7 @@ repl_backlog_first_byte_offset:206989083 repl_backlog_histlen:1048576 ``` -When it's a `replica`, you will see details of the primary connection and if +When it's a `replica`, you see details of the primary connection and if its `up` or `down`: ```plaintext @@ -138,7 +138,7 @@ there may be something wrong with your configuration files or it can be related to [this upstream issue](https://github.com/redis/redis-rb/issues/531). You must make sure that `resque.yml` and `sentinel.conf` are configured correctly, -otherwise `redis-rb` will not work properly. +otherwise `redis-rb` does not work properly. The `master-group-name` (`gitlab-redis`) defined in (`sentinel.conf`) **must** be used as the hostname in GitLab (`resque.yml`): diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index 99272cdd226..a64828be4be 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -38,7 +38,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -153,6 +153,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -502,7 +503,7 @@ cluster to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -1271,7 +1272,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). @@ -1351,6 +1352,9 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + NOTE: + You can't remove the `default` entry from `virtual_storages` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md @@ -2202,7 +2206,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -2328,7 +2332,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index c356c7224cb..1a774603863 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -10,7 +10,7 @@ This page describes GitLab reference architecture for up to 1,000 users. For a full list of reference architectures, see [Available reference architectures](index.md#available-reference-architectures). -If you need to serve up to 1,000 users and you don't have strict availability +If you are serving up to 1,000 users and you don't have strict availability requirements, a single-node solution with [frequent backups](index.md#automated-backups) is appropriate for many organizations. @@ -83,6 +83,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -93,7 +94,8 @@ swap on your server, even if you currently have enough available memory. Having swap helps to reduce the chance of errors occurring if your available memory changes. We also recommend configuring the kernel's swappiness setting to a lower value (such as `10`) to make the most of your memory, while still having -the swap available when needed. +the swap available when needed. View the +[Memory requirements](../../install/requirements.md#memory) for details. ## Setup instructions @@ -121,4 +123,4 @@ components are deployed in Kubernetes via our official [Helm Charts](https://doc and _stateful_ components are deployed in compute VMs with Omnibus. The [2k GitLab Cloud Native Hybrid](2k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) (non HA) and [3k GitLab Cloud Native Hybrid](3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) (HA) reference architectures are the smallest we recommend in Kubernetes. -For environments that need to serve less users, you can lower the node specs. Depending on your user count, you can lower all suggested node specs as desired. However, it's recommended that you don't go lower than the [general requirements](../../install/requirements.md). +For environments that serve fewer users, you can lower the node specs. Depending on your user count, you can lower all suggested node specs as desired. However, it's recommended that you don't go lower than the [general requirements](../../install/requirements.md). diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 367654b8e59..5b1e8bfc16b 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -38,7 +38,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -153,6 +153,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -505,7 +506,7 @@ cluster to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -1277,7 +1278,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1355,6 +1356,9 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + NOTE: + You can't remove the `default` entry from `virtual_storages` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md @@ -2206,7 +2210,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -2326,7 +2330,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 3b9a8f966c8..df84bf0402d 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -31,7 +31,7 @@ For a full list of reference architectures, see | NFS server (non-Gitaly) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run as reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run as reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -89,6 +89,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -246,7 +247,7 @@ to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -444,6 +445,9 @@ To configure the Gitaly server, on the server node you want to use for Gitaly: 1. Edit the Gitaly server node's `/etc/gitlab/gitlab.rb` file to configure storage paths, enable the network listener, and to configure the token: + NOTE: + You can't remove the `default` entry from `git_data_dirs` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/advanced/external-gitaly/external-omnibus-gitaly.md#configure-omnibus-gitlab @@ -916,7 +920,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -1027,7 +1031,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index a43d33a4d4a..6a70739ca31 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -47,7 +47,7 @@ For a full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -159,6 +159,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -787,7 +788,7 @@ cluster to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -1217,7 +1218,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). @@ -1295,6 +1296,9 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + NOTE: + You can't remove the `default` entry from `virtual_storages` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md @@ -2141,7 +2145,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -2285,7 +2289,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index 316969dfacc..0d0e44e2364 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -38,7 +38,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -153,6 +153,7 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. @@ -511,7 +512,7 @@ cluster to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -1284,7 +1285,7 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Examples of the above could include [Google's Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) or [Amazon RDS](https://aws.amazon.com/rds/). @@ -1364,6 +1365,9 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + NOTE: + You can't remove the `default` entry from `virtual_storages` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md @@ -2222,7 +2226,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -2342,7 +2346,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index ef44d688f31..ef2e48baa33 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -44,7 +44,7 @@ costly-to-operate environment by using the <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -156,13 +156,14 @@ As a general guidance, GitLab should run on most infrastructure such as reputabl Be aware of the following specific call outs: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Database for PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/#:~:text=Azure%20Database%20for%20PostgreSQL%20is,high%20availability%2C%20and%20dynamic%20scalability.) is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to known performance issues or missing features. - [Azure Blob Storage](https://docs.microsoft.com/en-us/azure/storage/blobs/) is recommended to be configured with [Premium accounts](https://docs.microsoft.com/en-us/azure/storage/blobs/storage-blob-block-blob-premium) to ensure consistent performance. ### Praefect PostgreSQL It's worth noting that at this time [Praefect requires its own database server](../gitaly/praefect.md#postgresql) and -that to achieve full High Availability a third-party PostgreSQL database solution will be required. +that to achieve full High Availability a third-party PostgreSQL database solution is required. We hope to offer a built in solutions for these restrictions in the future but in the meantime a non HA PostgreSQL server can be set up via Omnibus GitLab, which the above specs reflect. Refer to the following issues for more information: [`omnibus-gitlab#5919`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5919) & [`gitaly#3398`](https://gitlab.com/gitlab-org/gitaly/-/issues/3398). @@ -232,7 +233,7 @@ The following list includes descriptions of each server and its assigned IP: ## Configure the external load balancer -In a multi-node GitLab configuration, you'll need a load balancer to route +In a multi-node GitLab configuration, you need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or its exact configuration is beyond the scope of GitLab documentation. We assume that if you're managing multi-node systems like GitLab, you already have a load @@ -245,7 +246,7 @@ This architecture has been tested and validated with [HAProxy](https://www.hapro as the load balancer. Although other load balancers with similar feature sets could also be used, those load balancers have not been validated. -The next question is how you will handle SSL in your environment. +The next question is how you handle SSL in your environment. There are several different options: - [The application node terminates SSL](#application-node-terminates-ssl). @@ -257,8 +258,8 @@ There are several different options: ### Application node terminates SSL Configure your load balancer to pass connections on port 443 as `TCP` rather -than `HTTP(S)` protocol. This will pass the connection to the application node's -NGINX service untouched. NGINX will have the SSL certificate and listen on port 443. +than `HTTP(S)` protocol. This passes the connection to the application node's +NGINX service untouched. NGINX has the SSL certificate and listen on port 443. See the [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. @@ -266,10 +267,10 @@ for details on managing SSL certificates and configuring NGINX. ### Load balancer terminates SSL without backend SSL Configure your load balancer to use the `HTTP(S)` protocol rather than `TCP`. -The load balancer will then be responsible for managing SSL certificates and +The load balancer is then responsible for managing SSL certificates and terminating SSL. -Since communication between the load balancer and GitLab will not be secure, +Since communication between the load balancer and GitLab is not secure, there is some additional configuration needed. See the [NGINX proxied SSL documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#supporting-proxied-ssl) for details. @@ -277,12 +278,12 @@ for details. ### Load balancer terminates SSL with backend SSL Configure your load balancers to use the 'HTTP(S)' protocol rather than 'TCP'. -The load balancers will be responsible for managing SSL certificates that -end users will see. +The load balancers are responsible for managing SSL certificates that +end users see. -Traffic will also be secure between the load balancers and NGINX in this +Traffic is also secure between the load balancers and NGINX in this scenario. There is no need to add configuration for proxied SSL since the -connection will be secure all the way. However, configuration will need to be +connection is secure all the way. However, configuration needs to be added to GitLab to configure SSL certificates. See [NGINX HTTPS documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https) for details on managing SSL certificates and configuring NGINX. @@ -292,7 +293,7 @@ for details on managing SSL certificates and configuring NGINX. Ensure the external load balancer only routes to working services with built in monitoring endpoints. The [readiness checks](../../user/admin_area/monitoring/health_check.md) all require [additional configuration](../monitoring/ip_whitelist.md) -on the nodes being checked, otherwise, the external load balancer will not be able to +on the nodes being checked, otherwise, the external load balancer is not able to connect. ### Ports @@ -311,11 +312,11 @@ The basic ports to be used are shown in the table below. to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. -- (*2*): When using HTTPS protocol for port 443, you will need to add an SSL +- (*2*): When using HTTPS protocol for port 443, you need to add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. -If you're using GitLab Pages with custom domain support you will need some +If you're using GitLab Pages with custom domain support you need some additional port configurations. GitLab Pages requires a separate virtual IP address. Configure DNS to point the `pages_external_url` from `/etc/gitlab/gitlab.rb` at the new virtual IP address. See the @@ -337,7 +338,7 @@ GitLab Pages requires a separate virtual IP address. Configure DNS to point the Some organizations have policies against opening SSH port 22. In this case, it may be helpful to configure an alternate SSH hostname that allows users -to use SSH on port 443. An alternate SSH hostname will require a new virtual IP address +to use SSH on port 443. An alternate SSH hostname requires a new virtual IP address compared to the other GitLab HTTP configuration above. Configure DNS for an alternate SSH hostname such as `altssh.gitlab.example.com`. @@ -359,7 +360,7 @@ such as connections to [PgBouncer](#configure-pgbouncer) and [Praefect](#configu It's a separate node from the External Load Balancer and shouldn't have any access externally. -The following IP will be used as an example: +The following IP is used as an example: - `10.6.0.40`: Internal Load Balancer @@ -433,8 +434,8 @@ You are highly encouraged to read the [Redis Sentinel](https://redis.io/topics/s before configuring Redis with GitLab to fully understand the topology and architecture. -In this section, you'll be guided through configuring an external Redis instance -to be used with GitLab. The following IPs will be used as an example: +In this section, you are guided through configuring an external Redis instance +to be used with GitLab. The following IPs are used as an example: - `10.6.0.61`: Redis Primary - `10.6.0.62`: Redis Replica 1 @@ -442,7 +443,7 @@ to be used with GitLab. The following IPs will be used as an example: ### Provide your own Redis instance -Managed Redis from cloud providers such as AWS ElastiCache will work. If these +Managed Redis from cloud providers such as AWS ElastiCache works. If these services support high availability, be sure it is **not** the Redis Cluster type. Redis version 5.0 or higher is required, as this is what ships with @@ -451,7 +452,7 @@ do not support an optional count argument to SPOP which is now required for [Merge Trains](../../ci/pipelines/merge_trains.md). Note the Redis node's IP address or hostname, port, and password (if required). -These will be necessary when configuring the +These are necessary when configuring the [GitLab application servers](#configure-gitlab-rails) later. ### Standalone Redis using Omnibus GitLab @@ -617,8 +618,8 @@ You can specify multiple roles, like sentinel and Redis, as: [roles](https://docs.gitlab.com/omnibus/roles/). These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after -a failover, as the nodes will be managed by the [Sentinels](#configure-consul-and-sentinel), and even after a -`gitlab-ctl reconfigure`, they will get their configuration restored by +a failover, as the nodes are managed by the [Sentinels](#configure-consul-and-sentinel), and even after a +`gitlab-ctl reconfigure`, they get their configuration restored by the same Sentinels. Advanced [configuration options](https://docs.gitlab.com/omnibus/settings/redis.html) @@ -633,7 +634,7 @@ are supported and can be added if needed. ## Configure Consul and Sentinel Now that the Redis servers are all set up, let's configure the Sentinel -servers. The following IPs will be used as an example: +servers. The following IPs are used as an example: - `10.6.0.11`: Consul/Sentinel 1 - `10.6.0.12`: Consul/Sentinel 2 @@ -647,7 +648,7 @@ clients to report `NOAUTH Authentication required.`. To configure the Sentinel: -1. SSH in to the server that will host Consul/Sentinel. +1. SSH in to the server that hosts Consul/Sentinel. 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package of your choice. Be sure to both follow _only_ installation steps 1 and 2 on the page, and to select the correct Omnibus GitLab package, with the same version @@ -776,7 +777,7 @@ run: sentinel: (pid 30098) 76832s; run: log: (pid 29704) 76850s ## Configure PostgreSQL -In this section, you'll be guided through configuring a highly available PostgreSQL +In this section, you are guided through configuring a highly available PostgreSQL cluster to be used with GitLab. ### Provide your own PostgreSQL instance @@ -784,7 +785,7 @@ cluster to be used with GitLab. If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). If you use a cloud-managed service, or provide your own PostgreSQL: @@ -813,7 +814,7 @@ replication and failover requires: A local PgBouncer service to be configured on each PostgreSQL node. Note that this is separate from the main PgBouncer cluster that tracks the primary. -The following IPs will be used as an example: +The following IPs are used as an example: - `10.6.0.31`: PostgreSQL primary - `10.6.0.32`: PostgreSQL secondary 1 @@ -828,8 +829,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. #### PostgreSQL nodes 1. SSH in to one of the PostgreSQL nodes. -1. Generate a password hash for the PostgreSQL username/password pair. This assumes you will use the default - username of `gitlab` (recommended). The command will request a password +1. Generate a password hash for the PostgreSQL username/password pair. This assumes you use the default + username of `gitlab` (recommended). The command requests a password and confirmation. Use the value that is output by this command in the next step as the value of `<postgresql_password_hash>`: @@ -837,8 +838,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 gitlab ``` -1. Generate a password hash for the PgBouncer username/password pair. This assumes you will use the default - username of `pgbouncer` (recommended). The command will request a password +1. Generate a password hash for the PgBouncer username/password pair. This assumes you use the default + username of `pgbouncer` (recommended). The command requests a password and confirmation. Use the value that is output by this command in the next step as the value of `<pgbouncer_password_hash>`: @@ -846,8 +847,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 pgbouncer ``` -1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you will use the default - username of `gitlab_replicator` (recommended). The command will request a password +1. Generate a password hash for the PostgreSQL replication username/password pair. This assumes you use the default + username of `gitlab_replicator` (recommended). The command requests a password and a confirmation. Use the value that is output by this command in the next step as the value of `<postgresql_replication_password_hash>`: @@ -855,8 +856,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. sudo gitlab-ctl pg-password-md5 gitlab_replicator ``` -1. Generate a password hash for the Consul database username/password pair. This assumes you will use the default - username of `gitlab-consul` (recommended). The command will request a password +1. Generate a password hash for the Consul database username/password pair. This assumes you use the default + username of `gitlab-consul` (recommended). The command requests a password and confirmation. Use the value that is output by this command in the next step as the value of `<consul_password_hash>`: @@ -935,7 +936,7 @@ in the second step, do not supply the `EXTERNAL_URL` value. # END user configuration ``` -PostgreSQL, with Patroni managing its failover, will default to use `pg_rewind` by default to handle conflicts. +PostgreSQL, with Patroni managing its failover, defaults to use `pg_rewind` by default to handle conflicts. Like most failover handling methods, this has a small chance of leading to data loss. Learn more about the various [Patroni replication methods](../postgresql/replication_and_failover.md#selecting-the-appropriate-patroni-replication-method). @@ -987,7 +988,7 @@ If the 'State' column for any node doesn't say "running", check the Now that the PostgreSQL servers are all set up, let's configure PgBouncer for tracking and handling reads/writes to the primary database. -The following IPs will be used as an example: +The following IPs are used as an example: - `10.6.0.21`: PgBouncer 1 - `10.6.0.22`: PgBouncer 2 @@ -1111,9 +1112,9 @@ The recommended cluster setup includes the following components: - 1 Praefect PostgreSQL node: Database server for Praefect. A third-party solution is required for Praefect database connections to be made highly available. - 1 load balancer: A load balancer is required for Praefect. The - [internal load balancer](#configure-the-internal-load-balancer) will be used. + [internal load balancer](#configure-the-internal-load-balancer) is used. -This section will detail how to configure the recommended standard setup in order. +This section details how to configure the recommended standard setup in order. For more advanced setups refer to the [standalone Gitaly Cluster documentation](../gitaly/praefect.md). ### Configure Praefect PostgreSQL @@ -1125,7 +1126,7 @@ A built-in solution is being [worked on](https://gitlab.com/gitlab-org/omnibus-g #### Praefect non-HA PostgreSQL standalone using Omnibus GitLab -The following IPs will be used as an example: +The following IPs are used as an example: - `10.6.0.141`: Praefect PostgreSQL @@ -1137,8 +1138,8 @@ in the second step, do not supply the `EXTERNAL_URL` value. 1. SSH in to the Praefect PostgreSQL node. 1. Create a strong password to be used for the Praefect PostgreSQL user. Take note of this password as `<praefect_postgresql_password>`. -1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you will use the default - username of `praefect` (recommended). The command will request the password `<praefect_postgresql_password>` +1. Generate the password hash for the Praefect PostgreSQL username/password pair. This assumes you use the default + username of `praefect` (recommended). The command requests the password `<praefect_postgresql_password>` and confirmation. Use the value that is output by this command in the next step as the value of `<praefect_postgresql_password_hash>`: @@ -1215,13 +1216,13 @@ you are using Geo, where separate database instances are required for handling r In this setup, the specs of the main database setup shouldn't need to be changed as the impact should be minimal. -A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). +A reputable provider or solution should be used for this. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Once the database is set up, follow the [post configuration](#praefect-postgresql-post-configuration). #### Praefect PostgreSQL post-configuration -After the Praefect PostgreSQL server has been set up, you'll then need to configure the user and database for Praefect to use. +After the Praefect PostgreSQL server has been set up, you then need to configure the user and database for Praefect to use. We recommend the user be named `praefect` and the database `praefect_production`, and these can be configured as standard in PostgreSQL. The password for the user is the same as the one you configured earlier as `<praefect_postgresql_password>`. @@ -1274,12 +1275,12 @@ Praefect requires several secret tokens to secure communications across the Clus Gitaly Cluster nodes are configured in Praefect via a `virtual storage`. Each storage contains the details of each Gitaly node that makes up the cluster. Each storage is also given a name -and this name is used in several areas of the configuration. In this guide, the name of the storage will be +and this name is used in several areas of the configuration. In this guide, the name of the storage is `default`. Also, this guide is geared towards new installs, if upgrading an existing environment to use Gitaly Cluster, you may need to use a different name. Refer to the [Praefect documentation](../gitaly/praefect.md#praefect) for more information. -The following IPs will be used as an example: +The following IPs are used as an example: - `10.6.0.131`: Praefect 1 - `10.6.0.132`: Praefect 2 @@ -1293,6 +1294,9 @@ To configure the Praefect nodes, on each one: on the page. 1. Edit the `/etc/gitlab/gitlab.rb` file to configure Praefect: + NOTE: + You can't remove the `default` entry from `virtual_storages` because [GitLab requires it](../gitaly/configure_gitaly.md#gitlab-requires-a-default-repository-storage). + <!-- Updates to example must be made at: - https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/administration/gitaly/praefect.md @@ -1429,7 +1433,7 @@ For configuring Gitaly you should note the following: - `git_data_dirs` should be configured to reflect the storage path for the specific Gitaly node - `auth_token` should be the same as `praefect_internal_token` -The following IPs will be used as an example: +The following IPs are used as an example: - `10.6.0.91`: Gitaly 1 - `10.6.0.92`: Gitaly 2 @@ -1811,7 +1815,7 @@ On each node perform the following: ``` 1. Specify the necessary NFS mounts in `/etc/fstab`. - The exact contents of `/etc/fstab` will depend on how you chose + The exact contents of `/etc/fstab` depends on how you chose to configure your NFS server. See the [NFS documentation](../nfs.md) for examples and the various options. @@ -1827,9 +1831,9 @@ On each node perform the following: on the page. 1. Create or edit `/etc/gitlab/gitlab.rb` and use the following configuration. To maintain uniformity of links across nodes, the `external_url` - on the application server should point to the external URL that users will use + on the application server should point to the external URL that users use to access GitLab. This would be the URL of the [external load balancer](#configure-the-external-load-balancer) - which will route traffic to the GitLab application server: + which routes traffic to the GitLab application server: ```ruby external_url 'https://gitlab.example.com' @@ -1999,7 +2003,7 @@ On each node perform the following: When you specify `https` in the `external_url`, as in the previous example, GitLab expects that the SSL certificates are in `/etc/gitlab/ssl/`. If the -certificates aren't present, NGINX will fail to start. For more information, see +certificates aren't present, NGINX fails to start. For more information, see the [NGINX documentation](https://docs.gitlab.com/omnibus/settings/nginx.html#enable-https). ### GitLab Rails post-configuration @@ -2141,7 +2145,7 @@ on what features you intend to use: |Object storage type|Supported by consolidated configuration?| |-------------------|----------------------------------------| -| [Backups](../../raketasks/backup_restore.md#uploading-backups-to-a-remote-cloud-storage) | No | +| [Backups](../../raketasks/backup_gitlab.md#uploading-backups-to-a-remote-cloud-storage) | No | | [Job artifacts](../job_artifacts.md#using-object-storage) including archived job logs | Yes | | [LFS objects](../lfs/index.md#storing-lfs-objects-in-remote-object-storage) | Yes | | [Uploads](../uploads.md#using-object-storage) | Yes | @@ -2260,7 +2264,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work, however Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. [Google Cloud SQL](https://cloud.google.com/sql/docs/postgres/high-availability#normal) and [Amazon RDS](https://aws.amazon.com/rds/) are known to work. However, Amazon Aurora is **incompatible** with load balancing enabled by default in [14.4.0](../../update/index.md#1440), and Azure Database for PostgreSQL is **not recommended** due to [performance issues](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61). Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However, Consul is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS ElastiCache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/index.md b/doc/administration/reference_architectures/index.md index d9fa6c7bdd9..e996acb1157 100644 --- a/doc/administration/reference_architectures/index.md +++ b/doc/administration/reference_architectures/index.md @@ -296,7 +296,7 @@ The following table details the cost to run the different reference architecture NOTE: The following lists are non exhaustive. Generally, other cloud providers not listed -here will likely work with the same specs, but this hasn't been validated. +here likely work with the same specs, but this hasn't been validated. Additionally, when it comes to other cloud provider services not listed here, it's advised to be cautious as each implementation can be notably different and should be tested thoroughly before production use. @@ -366,9 +366,9 @@ Additionally, the following cloud provider services are validated and supported The following specific cloud provider services have been found to have issues in terms of either functionality or performance. As such, they either have caveats that should be considered or are not recommended: +- [Amazon Aurora](https://aws.amazon.com/rds/aurora/) is incompatible. See [14.4.0](../../update/index.md#1440) for more details. - [Azure Blob Storage](https://azure.microsoft.com/en-gb/services/storage/blobs/) has been found to have performance limits that can impact production use at certain times. For larger Reference Architectures the service may not be sufficient for production use and an alternative is recommended for use instead. - [Azure Database for PostgreSQL Server](https://azure.microsoft.com/en-gb/services/postgresql/#overview) (Single / Flexible) is not recommended for use due to notable performance issues or missing functionality. -- [AWS Aurora Database](https://aws.amazon.com/rds/aurora/) is not recommended due to compatibility issues. NOTE: As a general rule we unfortunately don't recommend Azure Services at this time. @@ -389,7 +389,7 @@ most complex: As you implement these components, begin with a single server and then do backups. Only after completing the first server should you proceed to the next. -Also, not implementing extra servers for GitLab doesn't necessarily mean that you'll have +Also, not implementing extra servers for GitLab doesn't necessarily mean that you have more downtime. Depending on your needs and experience level, single servers can have more actual perceived uptime for your users. @@ -401,7 +401,7 @@ have more actual perceived uptime for your users. This solution is appropriate for many teams that have the default GitLab installation. With automatic backups of the GitLab repositories, configuration, and the database, this can be an optimal solution if you don't have strict requirements. -[Automated backups](../../raketasks/backup_restore.md#configuring-cron-to-make-daily-backups) +[Automated backups](../../raketasks/backup_gitlab.md#configuring-cron-to-make-daily-backups) is the least complex to setup. This provides a point-in-time recovery of a predetermined schedule. ### Traffic load balancer **(PREMIUM SELF)** @@ -410,7 +410,7 @@ is the least complex to setup. This provides a point-in-time recovery of a prede > - Required domain knowledge: HAProxy, shared storage, distributed systems This requires separating out GitLab into multiple application nodes with an added -[load balancer](../load_balancer.md). The load balancer will distribute traffic +[load balancer](../load_balancer.md). The load balancer distributes traffic across GitLab application nodes. Meanwhile, each application node connects to a shared file server and database systems on the back end. This way, if one of the application servers fails, the workflow is not interrupted. @@ -434,7 +434,7 @@ to any of the [available reference architectures](#available-reference-architect GitLab supports [zero-downtime upgrades](../../update/zero_downtime.md). Single GitLab nodes can be updated with only a [few minutes of downtime](../../update/index.md#upgrade-based-on-installation-method). To avoid this, we recommend to separate GitLab into several application nodes. -As long as at least one of each component is online and capable of handling the instance's usage load, your team's productivity will not be interrupted during the update. +As long as at least one of each component is online and capable of handling the instance's usage load, your team's productivity is not interrupted during the update. ### Automated database failover **(PREMIUM SELF)** @@ -459,8 +459,8 @@ that can also be promoted in case of disaster. ## Deviating from the suggested reference architectures As a general guideline, the further away you move from the Reference Architectures, -the harder it will be get support for it. With any deviation, you're introducing -a layer of complexity that will add challenges to finding out where potential +the harder it is to get support for it. With any deviation, you're introducing +a layer of complexity that adds challenges to finding out where potential issues might lie. The reference architectures use the official GitLab Linux packages (Omnibus @@ -474,7 +474,7 @@ However, it is still an additional layer and may still add some support complexi Other technologies, like [Docker swarm](https://docs.docker.com/engine/swarm/) are not officially supported, but can be implemented at your own risk. In that -case, GitLab Support will not be able to help you. +case, GitLab Support is not able to help you. ## Supported modifications for lower user count HA reference architectures diff --git a/doc/administration/reference_architectures/troubleshooting.md b/doc/administration/reference_architectures/troubleshooting.md index 2d10d05da37..6e0aaacd598 100644 --- a/doc/administration/reference_architectures/troubleshooting.md +++ b/doc/administration/reference_architectures/troubleshooting.md @@ -13,7 +13,7 @@ the [reference architectures](index.md#reference-architectures). ### S3 API compatibility issues -Not all S3 providers [are fully compatible](../../raketasks/backup_restore.md#other-s3-providers) +Not all S3 providers [are fully compatible](../../raketasks/backup_gitlab.md#other-s3-providers) with the Fog library that GitLab uses. Symptoms include: ```plaintext @@ -54,20 +54,20 @@ This can result in some of the following problems: - If GitLab is using non-secure HTTP to access the object storage, clients may generate `https->http` downgrade errors and refuse to process the redirect. The solution to this -is for GitLab to use HTTPS. LFS, for example, will generate this error: +is for GitLab to use HTTPS. LFS, for example, generates this error: ```plaintext LFS: lfsapi/client: refusing insecure redirect, https->http ``` -- Clients will need to trust the certificate authority that issued the object storage +- Clients must trust the certificate authority that issued the object storage certificate, or may return common TLS errors such as: ```plaintext x509: certificate signed by unknown authority ``` -- Clients will need network access to the object storage. Errors that might result +- Clients need network access to the object storage. Errors that might result if this access is not in place include: ```plaintext @@ -113,7 +113,7 @@ You can check if everything is correct by connecting to each server using /opt/gitlab/embedded/bin/redis-cli -h <redis-host-or-ip> -a '<redis-password>' info replication ``` -When connected to a `Primary` Redis, you will see the number of connected +When connected to a `Primary` Redis, you see the number of connected `replicas`, and a list of each with connection details: ```plaintext @@ -128,7 +128,7 @@ repl_backlog_first_byte_offset:206989083 repl_backlog_histlen:1048576 ``` -When it's a `replica`, you will see details of the primary connection and if +When it's a `replica`, you see details of the primary connection and if its `up` or `down`: ```plaintext @@ -254,7 +254,7 @@ To start a session: sudo gitlab-ctl pgb-console ``` -The password you will be prompted for is the `pgbouncer_user_password` +The password you are prompted for is the `pgbouncer_user_password` To get some basic information about the instance, run @@ -315,7 +315,7 @@ sudo gitlab-ctl tail patroni ### Consul and PostgreSQL with Patroni changes not taking effect -Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it will not restart the services. However, not all changes can be activated by reloading. +Due to the potential impacts, `gitlab-ctl reconfigure` only reloads Consul and PostgreSQL, it does not restart the services. However, not all changes can be activated by reloading. To restart either service, run `gitlab-ctl restart consul` or `gitlab-ctl restart patroni` respectively. @@ -335,7 +335,7 @@ PG::ConnectionBad: ERROR: pgbouncer cannot connect to server The problem may be that your PgBouncer node's IP address is not included in the `trust_auth_cidr_addresses` setting in `/etc/gitlab/gitlab.rb` on the database nodes. -You can confirm that this is the issue by checking the PostgreSQL log on the master +You can confirm that this is the issue by checking the PostgreSQL log on the primary database node. If you see the following error then `trust_auth_cidr_addresses` is the problem. diff --git a/doc/administration/reply_by_email_postfix_setup.md b/doc/administration/reply_by_email_postfix_setup.md index 2652ad20329..0fbb4562995 100644 --- a/doc/administration/reply_by_email_postfix_setup.md +++ b/doc/administration/reply_by_email_postfix_setup.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Set up Postfix for incoming email **(FREE SELF)** -This document will take you through the steps of setting up a basic Postfix mail +This document takes you through the steps of setting up a basic Postfix mail server with IMAP authentication on Ubuntu, to be used with [incoming email](incoming_email.md). The instructions make the assumption that you are using the email address `incoming@gitlab.example.com`, that is, username `incoming` on host `gitlab.example.com`. Don't forget to change it to your actual host when executing the example code snippets. @@ -46,7 +46,7 @@ The instructions make the assumption that you are using the email address `incom sudo passwd incoming ``` - Be sure not to forget this, you'll need it later. + Be sure not to forget this, you will need it later. ## Test the out-of-the-box setup diff --git a/doc/administration/repository_checks.md b/doc/administration/repository_checks.md index c2233f70a9a..8cfefdb9b56 100644 --- a/doc/administration/repository_checks.md +++ b/doc/administration/repository_checks.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Repository checks **(FREE SELF)** diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index 3acaad8231e..f001ba35bca 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Repository storage **(FREE SELF)** @@ -169,7 +168,6 @@ NOTE: If all storage weights are `0` (for example, when `default` does not exist), GitLab attempts to create new repositories on `default`, regardless of the configuration or if `default` exists. See [the tracking issue](https://gitlab.com/gitlab-org/gitlab/-/issues/36175) for more information. -information. ## Move repositories diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index 1a593c8c6bd..7ce629b5d71 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto --- # Repository storage types **(FREE SELF)** diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index 06b6cbd4271..6ac031ea6bd 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -31,7 +31,7 @@ GitLab Rails application (Puma) as well as the other components, like: ### Omnibus GitLab restart There may be times in the documentation where you are asked to _restart_ -GitLab. In that case, you need to run the following command: +GitLab. To restart, run the following command: ```shell sudo gitlab-ctl restart diff --git a/doc/administration/server_hooks.md b/doc/administration/server_hooks.md index a83a17d6d52..1a47dc4ccf2 100644 --- a/doc/administration/server_hooks.md +++ b/doc/administration/server_hooks.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto disqus_identifier: 'https://docs.gitlab.com/ee/administration/custom_hooks.html' --- @@ -27,9 +26,9 @@ alternatives to server hooks include: [Geo](geo/index.md) doesn't replicate server hooks to secondary nodes. -## Create a server hook for a single repository +## Create server hooks for a repository -To create a server hook for a single repository: +To create server hooks for a repository: 1. On the top bar, select **Menu > Admin**. 1. Go to **Overview > Projects** and select the project you want to add a server hook to. @@ -41,16 +40,21 @@ To create a server hook for a single repository: - For Omnibus GitLab installations, the path is usually `/var/opt/gitlab/git-data/repositories/<group>/<project>.git`. - For an installation from source, the path is usually `/home/git/repositories/<group>/<project>.git`. 1. On the file system, create a new directory in the correct location called `custom_hooks`. -1. In the new `custom_hooks` directory, create a file with a name that matches the hook type. For example, for a - `pre-receive` server hook, the filename should be `pre-receive` with no extension. -1. Make the server hook file executable and ensure that it's owned by the Git user. +1. In the new `custom_hooks` directory: + - To create a single server hook, create a file with a name that matches the hook type. For example, for a + `pre-receive` server hook, the filename should be `pre-receive` with no extension. + - To create many server hooks, create a directory for the hooks that matches the hook type. For example, for a + `pre-receive` server hook, the directory name should be `pre-receive.d`. Put the files for the hook in that directory. +1. Make the server hook files executable and ensure that they are owned by the Git user. 1. Write the code to make the server hook function as expected. Server hooks can be in any programming language. Ensure the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`. +1. Make the hook file executable, ensure that it's owned by the Git user, and ensure it does not match the backup file + pattern (`*~`). If the server hook code is properly implemented, it should execute when the Git hook is next triggered. -## Create a global server hook for all repositories +## Create global server hooks for all repositories To create a Git hook that applies to all repositories, set a global server hook. The default global server hook directory is in the GitLab Shell directory. Any server hook added there applies to all repositories, including: @@ -80,8 +84,11 @@ To use a different directory for global server hooks, set `custom_hooks_dir` in To create a global server hook for all repositories: 1. On the GitLab server, go to the configured global server hook directory. -1. Create a new directory in this location called `pre-receive.d`, `post-receive.d`, or `update.d`, depending on the type - of server hook. Any other names are ignored. +1. In the configured global server hook directory: + - To create a single server hook, create a file with a name that matches the hook type. For example, for a + `pre-receive` server hook, the filename should be `pre-receive` with no extension. + - To create many server hooks, create a directory for the hooks that matches the hook type. For example, for a + `pre-receive` server hook, the directory name should be `pre-receive.d`. Put the files for the hook in that directory. 1. Inside this new directory, add your server hook. Server hooks can be in any programming language. Ensure the [shebang](https://en.wikipedia.org/wiki/Shebang_(Unix)) at the top reflects the language type. For example, if the script is in Ruby the shebang is probably `#!/usr/bin/env ruby`. diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index 528ecf12df9..fc24c764330 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -6,19 +6,92 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Configure an external Sidekiq instance **(FREE SELF)** -You can configure an external Sidekiq instance by using the Sidekiq that's -bundled in the GitLab package. Sidekiq requires connection to the Redis, +You can configure an external Sidekiq instance by using the Sidekiq that's bundled in the GitLab package. Sidekiq requires connection to the Redis, PostgreSQL, and Gitaly instances. -## Required configuration +## Configure TCP access for PostgreSQL, Gitaly, and Redis -To configure Sidekiq: +By default, GitLab uses UNIX sockets and is not set up to communicate via TCP. To change this: + +1. Edit the `/etc/gitlab/gitlab.rb` file on your GitLab instance, and add the following: + + ```ruby + + ## PostgreSQL + + # Replace POSTGRESQL_PASSWORD_HASH with a generated md5 value + postgresql['sql_user_password'] = 'POSTGRESQL_PASSWORD_HASH' + postgresql['listen_address'] = '0.0.0.0' + postgresql['port'] = 5432 + + # Add the Sidekiq nodes to PostgreSQL's trusted addresses. + # In the following example, 10.10.1.30/32 is the private IP + # of the Sidekiq server. + postgresql['md5_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) + postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) + + ## Gitaly + + # Make Gitaly accept connections on all network interfaces + gitaly['listen_addr'] = "0.0.0.0:8075" + ## Set up the Gitaly token as a form of authentication since you are accessing Gitaly over the network + ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token + gitaly['auth_token'] = 'abc123secret' + praefect['auth_token'] = 'abc123secret' + gitlab_rails['gitaly_token'] = 'abc123secret' + + ## Redis configuration + + redis['bind'] = '0.0.0.0' + redis['port'] = 6379 + # Password to Authenticate Redis + redis['password'] = 'redis-password-goes-here' + gitlab_rails['redis_password'] = 'redis-password-goes-here' + + gitlab_rails['auto_migrate'] = false + ``` + +1. Run `reconfigure`: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +1. Restart the `PostgreSQL` server: + + ```shell + sudo gitlab-ctl restart postgresql + ``` + +1. After the restart, set `auto_migrate` to `true` or comment to use the default settings: + + ```ruby + gitlab_rails['auto_migrate'] = true + ``` + +1. Run `reconfigure` again: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +## Set up Sidekiq instance 1. SSH into the Sidekiq server. + +1. Confirm that you can access the PostgreSQL, Gitaly, and Redis ports: + + ```shell + telnet <GitLab host> 5432 # PostgreSQL + telnet <GitLab host> 8075 # Gitaly + telnet <GitLab host> 6379 # Redis + ``` + 1. [Download and install](https://about.gitlab.com/install/) the Omnibus GitLab package using steps 1 and 2. **Do not complete any other steps.** -1. Edit `/etc/gitlab/gitlab.rb` with the following information and make sure - to replace with your values: + +1. Copy the `/etc/gitlab/gitlab.rb` file from the GitLab instance and add the following settings. Make sure + to replace them with your values: <!-- Updates to example must be made at: @@ -59,16 +132,25 @@ Updates to example must be made at: ## external_url 'https://gitlab.example.com' + # Configure the gitlab-shell API callback URL. Without this, `git push` will + # fail. This can be your 'front door' GitLab URL or an internal load + # balancer. + gitlab_rails['internal_api_url'] = 'GITLAB_URL' + gitlab_shell['secret_token'] = 'SHELL_TOKEN' + ######################################## #### Redis ### ######################################## - ## Must be the same in every sentinel node - redis['master_name'] = 'gitlab-redis' - + ## Must be the same in every sentinel node. + redis['master_name'] = 'gitlab-redis' # Required if you have setup redis cluster ## The same password for Redis authentication you set up for the master node. redis['master_password'] = '<redis_master_password>' + ### If redis is running on the main Gitlab instance and you have opened the TCP port as above add the following + gitlab_rails['redis_host'] = '<gitlab_host>' + gitlab_rails['redis_port'] = 6379 + ####################################### ### Gitaly ### ####################################### @@ -77,8 +159,8 @@ Updates to example must be made at: ## https://docs.gitlab.com/ee/administration/gitaly/configure_gitaly.html#about-the-gitaly-token git_data_dirs({ "default" => { - "gitaly_address" => "tcp://gitaly:8075", - "gitaly_token" => "<gitaly_token>" + "gitaly_address" => "tcp://<gitlab_host>:8075", + "gitaly_token" => "<gitaly_token>" } }) @@ -90,12 +172,6 @@ Updates to example must be made at: gitlab_rails['db_host'] = '<database_host>' gitlab_rails['db_port'] = '5432' gitlab_rails['db_password'] = '<database_password>' - - # Add the Sidekiq nodes to PostgreSQL's trusted addresses. - # In the following example, 10.10.1.30/32 is the private IP - # of the Sidekiq server. - postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32) - ## Prevent database migrations from running on upgrade automatically gitlab_rails['auto_migrate'] = false @@ -112,13 +188,15 @@ Updates to example must be made at: sidekiq['max_concurrency'] = 10 ``` +1. Copy the `/etc/gitlab/gitlab-secrets.json` file from the GitLab instance and replace the file in the Sidekiq instance. + 1. Reconfigure GitLab: ```shell sudo gitlab-ctl reconfigure ``` -1. Restart the Sidekiq nodes after completing the process and finishing the database migrations. +1. Restart the Sidekiq instance after completing the process and finishing the database migrations. ## Configure multiple Sidekiq nodes with shared storage @@ -191,6 +269,26 @@ To configure the metrics server: sudo gitlab-ctl reconfigure ``` +### Enable HTTPS + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/364771) in GitLab 15.2. + +To serve metrics via HTTPS instead of HTTP, enable TLS in the exporter settings: + +1. Edit `/etc/gitlab/gitlab.rb` to add (or find and uncomment) the following lines: + + ```ruby + sidekiq['exporter_tls_enabled'] = true + sidekiq['exporter_tls_cert_path'] = "/path/to/certificate.pem" + sidekiq['exporter_tls_key_path'] = "/path/to/private-key.pem" + ``` + +1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +When TLS is enabled, the same `port` and `address` are used as described above. +The metrics server cannot serve both HTTP and HTTPS at the same time. + ## Configure health checks If you use health check probes to observe Sidekiq, enable the Sidekiq health check server. @@ -198,7 +296,7 @@ To make health checks available from `localhost:8092`: 1. Edit `/etc/gitlab/gitlab.rb`: - ```ruby + ```ruby sidekiq['health_checks_enabled'] = true sidekiq['health_checks_listen_address'] = "localhost" sidekiq['health_checks_listen_port'] = "8092" diff --git a/doc/administration/smime_signing_email.md b/doc/administration/smime_signing_email.md index 943fd91c885..c9647129104 100644 --- a/doc/administration/smime_signing_email.md +++ b/doc/administration/smime_signing_email.md @@ -24,7 +24,7 @@ files must be provided: - Only RSA keys are supported. Optionally, you can also provide a bundle of CA certs (PEM-encoded) to be -included on each signature. This will typically be an intermediate CA. +included on each signature. This is typically an intermediate CA. WARNING: Be mindful of the access levels for your private keys and visibility to @@ -44,7 +44,7 @@ third parties. 1. Save the file and [reconfigure GitLab](restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -The key needs to be readable by the GitLab system user (`git` by default). +The key must be readable by the GitLab system user (`git` by default). **For installations from source:** @@ -67,7 +67,7 @@ The key needs to be readable by the GitLab system user (`git` by default). 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -The key needs to be readable by the GitLab system user (`git` by default). +The key must be readable by the GitLab system user (`git` by default). ### How to convert S/MIME PKCS #12 format to PEM encoding diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index 81ca1bda5d0..d5019f1aa4a 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -1,289 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../reference_architectures/troubleshooting.md' +remove_date: '2022-10-19' --- -# Debugging tips **(FREE SELF)** +This document was moved to [another location](../reference_architectures/troubleshooting.md). -Sometimes things don't work the way they should. Here are some tips on debugging issues out -in production. - -## Starting a Rails console session - -Troubleshooting and debugging your GitLab instance often requires a Rails console. - -Your type of GitLab installation determines how -[to start a rails console](../operations/rails_console.md). -See also: - -- [GitLab Rails Console Cheat Sheet](gitlab_rails_cheat_sheet.md). -- [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md). - -### Enabling Active Record logging - -You can enable output of Active Record debug logging in the Rails console -session by running: - -```ruby -ActiveRecord::Base.logger = Logger.new($stdout) -``` - -This will show information about database queries triggered by any Ruby code -you may run in the console. To turn off logging again, run: - -```ruby -ActiveRecord::Base.logger = nil -``` - -### Disabling database statement timeout - -You can disable the PostgreSQL statement timeout for the current Rails console -session by running: - -```ruby -ActiveRecord::Base.connection.execute('SET statement_timeout TO 0') -``` - -This change only affects the current Rails console session and will -not be persisted in the GitLab production environment or in the next Rails -console session. - -### Output Rails console session history - -If you'd like to output your Rails console command history in a format that's -easy to copy and save for future reference, you can run: - -```ruby -puts Readline::HISTORY.to_a -``` - -## Using the Rails runner - -If you need to run some Ruby code in the context of your GitLab production -environment, you can do so using the [Rails runner](https://guides.rubyonrails.org/command_line.html#rails-runner). When executing a script file, the script must be accessible by the `git` user. - -**For Omnibus installations** - -```shell -sudo gitlab-rails runner "RAILS_COMMAND" - -# Example with a two-line Ruby script -sudo gitlab-rails runner "user = User.first; puts user.username" - -# Example with a ruby script file (make sure to use the full path) -sudo gitlab-rails runner /path/to/script.rb -``` - -**For installations from source** - -```shell -sudo -u git -H bundle exec rails runner -e production "RAILS_COMMAND" - -# Example with a two-line Ruby script -sudo -u git -H bundle exec rails runner -e production "user = User.first; puts user.username" - -# Example with a ruby script file (make sure to use the full path) -sudo -u git -H bundle exec rails runner -e production /path/to/script.rb -``` - -## Mail not working - -A common problem is that mails are not being sent for some reason. Suppose you configured -an SMTP server, but you're not seeing mail delivered. Here's how to check the settings: - -1. Run a [Rails console](../operations/rails_console.md#starting-a-rails-console-session). - -1. Look at the ActionMailer `delivery_method` to make sure it matches what you - intended. If you configured SMTP, it should say `:smtp`. If you're using - Sendmail, it should say `:sendmail`: - - ```ruby - irb(main):001:0> ActionMailer::Base.delivery_method - => :smtp - ``` - -1. If you're using SMTP, check the mail settings: - - ```ruby - irb(main):002:0> ActionMailer::Base.smtp_settings - => {:address=>"localhost", :port=>25, :domain=>"localhost.localdomain", :user_name=>nil, :password=>nil, :authentication=>nil, :enable_starttls_auto=>true} - ``` - - In the example above, the SMTP server is configured for the local machine. If this is intended, you may need to check your local mail - logs (for example, `/var/log/mail.log`) for more details. - -1. Send a test message via the console. - - ```ruby - irb(main):003:0> Notify.test_email('youremail@email.com', 'Hello World', 'This is a test message').deliver_now - ``` - - If you do not receive an email and/or see an error message, then check - your mail server settings. - -## Advanced Issues - -For more advanced issues, `gdb` is a must-have tool for debugging issues. - -### The GNU Project Debugger (GDB) - -To install on Ubuntu/Debian: - -```shell -sudo apt-get install gdb -``` - -On CentOS: - -```shell -sudo yum install gdb -``` - -<!-- vale gitlab.Spelling = NO --> - -### rbtrace - -<!-- vale gitlab.Spelling = YES --> - -GitLab 11.2 ships with [`rbtrace`](https://github.com/tmm1/rbtrace), which -allows you to trace Ruby code, view all running threads, take memory dumps, -and more. However, this is not enabled by default. To enable it, define the -`ENABLE_RBTRACE` variable to the environment. For example, in Omnibus: - -```ruby -gitlab_rails['env'] = {"ENABLE_RBTRACE" => "1"} -``` - -Then reconfigure the system and restart Puma and Sidekiq. To run this -in Omnibus, run as root: - -```ruby -/opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/rbtrace -``` - -## Common Problems - -Many of the tips to diagnose issues below apply to many different situations. We'll use one -concrete example to illustrate what you can do to learn what is going wrong. - -### 502 Gateway Timeout after Unicorn spins at 100% CPU - -This error occurs when the Web server times out (default: 60 s) after not -hearing back from the Unicorn worker. If the CPU spins to 100% while this in -progress, there may be something taking longer than it should. - -To fix this issue, we first need to figure out what is happening. The -following tips are only recommended if you do not mind users being affected by -downtime. Otherwise skip to the next section. - -1. Load the problematic URL -1. Run `sudo gdb -p <PID>` to attach to the Puma process. -1. In the GDB window, type: - - ```plaintext - call (void) rb_backtrace() - ``` - -1. This forces the process to generate a Ruby backtrace. Check - `/var/log/gitlab/puma/puma_stderr.log` for the backtrace. For example, you may see: - - ```plaintext - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `block in start' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:33:in `loop' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:36:in `block (2 levels) in start' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:44:in `sample' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `sample_objects' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each_with_object' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:68:in `each' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `block in sample_objects' - from /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sampler.rb:69:in `name' - ``` - -1. To see the current threads, run: - - ```plaintext - thread apply all bt - ``` - -1. Once you're done debugging with `gdb`, be sure to detach from the process and exit: - - ```plaintext - detach - exit - ``` - -If the Puma process terminates before you are able to run these -commands, GDB will report an error. To buy more time, you can always raise the -Puma worker timeout. For omnibus users, you can edit `/etc/gitlab/gitlab.rb` and -increase it from 60 seconds to 600: - -```ruby -gitlab_rails['env'] = { - 'GITLAB_RAILS_RACK_TIMEOUT' => 600 -} -``` - -For source installations, set the environment variable. -Refer to [Puma Worker timeout](../operations/puma.md#change-the-worker-timeout). - -[Reconfigure](../restart_gitlab.md#omnibus-gitlab-reconfigure) GitLab for the changes to take effect. - -#### Troubleshooting without affecting other users - -The previous section attached to a running Unicorn process, and this may have -undesirable effects for users trying to access GitLab during this time. If you -are concerned about affecting others during a production system, you can run a -separate Rails process to debug the issue: - -1. Log in to your GitLab account. -1. Copy the URL that is causing problems (for example, `https://gitlab.com/ABC`). -1. Create a Personal Access Token for your user (User Settings -> Access Tokens). -1. Bring up the [GitLab Rails console.](../operations/rails_console.md#starting-a-rails-console-session) -1. At the Rails console, run: - - ```ruby - app.get '<URL FROM STEP 2>/?private_token=<TOKEN FROM STEP 3>' - ``` - - For example: - - ```ruby - app.get 'https://gitlab.com/gitlab-org/gitlab-foss/-/issues/1?private_token=123456' - ``` - -1. In a new window, run `top`. It should show this Ruby process using 100% CPU. Write down the PID. -1. Follow step 2 from the previous section on using GDB. - -### GitLab: API is not accessible - -This often occurs when GitLab Shell attempts to request authorization via the -[internal API](../../development/internal_api/index.md) (for example, `http://localhost:8080/api/v4/internal/allowed`), and -something in the check fails. There are many reasons why this may happen: - -1. Timeout connecting to a database (for example, PostgreSQL or Redis) -1. Error in Git hooks or push rules -1. Error accessing the repository (for example, stale NFS handles) - -To diagnose this problem, try to reproduce the problem and then see if there -is a Unicorn worker that is spinning via `top`. Try to use the `gdb` -techniques above. In addition, using `strace` may help isolate issues: - -```shell -strace -ttTfyyy -s 1024 -p <PID of puma worker> -o /tmp/puma.txt -``` - -If you cannot isolate which Unicorn worker is the issue, try to run `strace` -on all the Unicorn workers to see where the -[`/internal/allowed`](../../development/internal_api/index.md) endpoint gets stuck: - -```shell -ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/puma.txt -``` - -The output in `/tmp/puma.txt` may help diagnose the root cause. - -## More information - -- [Debugging Stuck Ruby Processes](https://newrelic.com/blog/best-practices/debugging-stuck-ruby-processes-what-to-do-before-you-kill-9) -- [Cheat sheet of using GDB and Ruby processes](gdb-stuck-ruby.txt) +<!-- This redirect file can be deleted after 2022-10-19. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/defcon.md b/doc/administration/troubleshooting/defcon.md index 292b4b13967..f2554f523f0 100644 --- a/doc/administration/troubleshooting/defcon.md +++ b/doc/administration/troubleshooting/defcon.md @@ -1,35 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: '../../ci/troubleshooting.md#disaster-recovery' +remove_date: '2022-10-04' --- -# Disaster recovery **(FREE SELF)** +This document was moved to [another location](../../ci/troubleshooting.md#disaster-recovery). -This document describes a feature that allows you to disable some important but computationally -expensive parts of the application to relieve stress on the database during an ongoing downtime. - -## `ci_queueing_disaster_recovery_disable_fair_scheduling` - -This feature flag, if temporarily enabled, disables fair scheduling on shared runners. -This can help to reduce system resource usage on the `jobs/request` endpoint -by significantly reducing the computations being performed. - -Side effects: - -- In case of a large backlog of jobs, the jobs are processed in the order - they were put in the system, instead of balancing the jobs across many projects. - -## `ci_queueing_disaster_recovery_disable_quota` - -This feature flag, if temporarily enabled, disables enforcing CI/CD minutes quota -on shared runners. This can help to reduce system resource usage on the -`jobs/request` endpoint by significantly reducing the computations being -performed. - -Side effects: - -- Projects which are out of quota will be run. This affects - only jobs created during the last hour, as prior jobs are canceled - by a periodic background worker (`StuckCiJobsWorker`). +<!-- This redirect file can be deleted after <2022-10-04>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/diagnostics_tools.md b/doc/administration/troubleshooting/diagnostics_tools.md index 6055746238f..faddf12b97d 100644 --- a/doc/administration/troubleshooting/diagnostics_tools.md +++ b/doc/administration/troubleshooting/diagnostics_tools.md @@ -8,7 +8,7 @@ type: reference # Diagnostics tools **(FREE SELF)** These are some of the diagnostics tools the GitLab Support team uses during troubleshooting. -They are listed here for transparency, and they may be useful for users with experience +They are listed here for transparency, and for users with experience with troubleshooting GitLab. If you are currently having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/) first, before attempting to use these tools. @@ -24,6 +24,6 @@ running on. [strace-parser](https://gitlab.com/wchandler/strace-parser) is a small tool to analyze and summarize raw `strace` data. -## kubesos +## `kubesos` The [`kubesos`](https://gitlab.com/gitlab-com/support/toolbox/kubesos/) utility retrieves GitLab cluster configuration and logs from GitLab Cloud Native chart deployments. diff --git a/doc/administration/troubleshooting/gdb-stuck-ruby.txt b/doc/administration/troubleshooting/gdb-stuck-ruby.txt deleted file mode 100644 index de8d704f9e3..00000000000 --- a/doc/administration/troubleshooting/gdb-stuck-ruby.txt +++ /dev/null @@ -1,142 +0,0 @@ -# Here's the script I'll use to demonstrate - it just loops forever: - -$ cat test.rb -#!/usr/bin/env ruby - -loop do - sleep 1 -end - -# Now, I'll start the script in the background, and redirect stdout and stderr -# to /dev/null: - -$ ruby ./test.rb >/dev/null 2>/dev/null & -[1] 1343 - -# Next, I'll grab the PID of the script (1343): - -$ ps aux | grep test.rb -vagrant 1343 0.0 0.4 3884 1652 pts/0 S 14:42 0:00 ruby ./test.rb -vagrant 1345 0.0 0.2 4624 852 pts/0 S+ 14:42 0:00 grep --color=auto test.rb - -# Now I start gdb. Note that I'm using sudo here. This may or may not be -# necessary in your setup. I'd try without sudo first, and fall back to adding -# it if the next step fails: - -$ sudo gdb -GNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04 -Copyright (C) 2012 Free Software Foundation, Inc. -License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> -This is free software: you are free to change and redistribute it. -There is NO WARRANTY, to the extent permitted by law. Type "show copying" -and "show warranty" for details. -This GDB was configured as "i686-linux-gnu". -For bug reporting instructions, please see: -<http://bugs.launchpad.net/gdb-linaro/>. - -# OK, now I'm in gdb, and I want to instruct it to attach to our Ruby process. -# I can do that using the 'attach' command, which takes a PID (the one we -# gathered above): - -(gdb) attach 1343 -Attaching to process 1343 -Reading symbols from /opt/vagrant_ruby/bin/ruby...done. -Reading symbols from /lib/i386-linux-gnu/librt.so.1...(no debugging symbols found)...done. -Loaded symbols for /lib/i386-linux-gnu/librt.so.1 -Reading symbols from /lib/i386-linux-gnu/libdl.so.2...(no debugging symbols found)...done. -Loaded symbols for /lib/i386-linux-gnu/libdl.so.2 -Reading symbols from /lib/i386-linux-gnu/libcrypt.so.1...(no debugging symbols found)...done. -Loaded symbols for /lib/i386-linux-gnu/libcrypt.so.1 -Reading symbols from /lib/i386-linux-gnu/libm.so.6...(no debugging symbols found)...done. -Loaded symbols for /lib/i386-linux-gnu/libm.so.6 -Reading symbols from /lib/i386-linux-gnu/libc.so.6...(no debugging symbols found)...done. -Loaded symbols for /lib/i386-linux-gnu/libc.so.6 -Reading symbols from /lib/i386-linux-gnu/libpthread.so.0...(no debugging symbols found)...done. -[Thread debugging using libthread_db enabled] -Using host libthread_db library "/lib/i386-linux-gnu/libthread_db.so.1". -Loaded symbols for /lib/i386-linux-gnu/libpthread.so.0 -Reading symbols from /lib/ld-linux.so.2...(no debugging symbols found)...done. -Loaded symbols for /lib/ld-linux.so.2 -0xb770c424 in __kernel_vsyscall () - -# Great, now gdb is attached to the target process. If the step above fails, try -# going back and running gdb under sudo. The next thing I want to do is gather -# C-level backtraces from all threads in the process. The following command -# stands for 'thread apply all backtrace': - -(gdb) t a a bt - -Thread 1 (Thread 0xb74d76c0 (LWP 1343)): -#0 0xb770c424 in __kernel_vsyscall () -#1 0xb75d7abd in select () from /lib/i386-linux-gnu/libc.so.6 -#2 0x08069c56 in rb_thread_wait_for (time=...) at eval.c:11376 -#3 0x080a20fd in rb_f_sleep (argc=1, argv=0xbf85f490) at process.c:1633 -#4 0x0805e0e2 in call_cfunc (argv=0xbf85f490, argc=1, len=-1, recv=3075299660, func=0x80a20b0 <rb_f_sleep>) - at eval.c:5778 -#5 rb_call0 (klass=3075304600, recv=3075299660, id=9393, oid=9393, argc=1, argv=0xbf85f490, body=0xb74c85a8, flags=2) - at eval.c:5928 -#6 0x0805e35d in rb_call (klass=3075304600, recv=3075299660, mid=9393, argc=1, argv=0xbf85f490, scope=1, - self=<optimized out>) at eval.c:6176 -#7 0x080651ec in rb_eval (self=3075299660, n=0xb74c4e1c) at eval.c:3521 -#8 0x0805c31c in rb_yield_0 (val=6, self=3075299660, klass=<optimized out>, flags=0, avalue=0) at eval.c:5095 -#9 0x0806a1e5 in loop_i () at eval.c:5227 -#10 0x08058dbd in rb_rescue2 (b_proc=0x806a1c0 <loop_i>, data1=0, r_proc=0, data2=0) at eval.c:5491 -#11 0x08058f28 in rb_f_loop () at eval.c:5252 -#12 0x0805e0c1 in call_cfunc (argv=0x0, argc=0, len=0, recv=3075299660, func=0x8058ef0 <rb_f_loop>) at eval.c:5781 -#13 rb_call0 (klass=3075304600, recv=3075299660, id=4121, oid=4121, argc=0, argv=0x0, body=0xb74d4dbc, flags=2) - at eval.c:5928 -#14 0x0805e35d in rb_call (klass=3075304600, recv=3075299660, mid=4121, argc=0, argv=0x0, scope=1, self=<optimized out>) - at eval.c:6176 -#15 0x080651ec in rb_eval (self=3075299660, n=0xb74c4dcc) at eval.c:3521 -#16 0x080662c6 in rb_eval (self=3075299660, n=0xb74c4de0) at eval.c:3236 -#17 0x08068ee4 in ruby_exec_internal () at eval.c:1654 -#18 0x08068f24 in ruby_exec () at eval.c:1674 -#19 0x0806b2cd in ruby_run () at eval.c:1684 -#20 0x08053771 in main (argc=2, argv=0xbf860204, envp=0xbf860210) at main.c:48 - -# C backtraces are sometimes sufficient, but often Ruby backtraces are necessary -# for debugging as well. Ruby has a built-in function called rb_backtrace() that -# we can use to dump out a Ruby backtrace, but it prints to stdout or stderr -# (depending on your Ruby version), which might have been redirected to a file -# or to /dev/null (as in our example) when the process started up. -# -# To get aroundt this, we'll do a little trick and redirect the target process's -# stdout and stderr to the current TTY, so that any output from the process -# will appear directly on our screen. - -# First, let's close the existing file descriptors for stdout and stderr -# (FD 1 and 2, respectively): -(gdb) call (void) close(1) -(gdb) call (void) close(2) - -# Next, we need to figure out the device name for the current TTY: -(gdb) shell tty -/dev/pts/0 - -# OK, now we can pass the device name obtained above to open() and attach -# file descriptors 1 and 2 back to the current TTY with these calls: - -(gdb) call (int) open("/dev/pts/0", 2, 0) -$1 = 1 -(gdb) call (int) open("/dev/pts/0", 2, 0) -$2 = 2 - -# Finally, we call rb_backtrace() in order to dump the Ruby backtrace: - -(gdb) call (void) rb_backtrace() - from ./test.rb:4:in `sleep' - from ./test.rb:4 - from ./test.rb:3:in `loop' - from ./test.rb:3 - -# And here's how we get out of gdb. Once you've quit, you'll probably want to -# clean up the stuck process by killing it. - -(gdb) quit -A debugging session is active. - - Inferior 1 [process 1343] will be detached. - -Quit anyway? (y or n) y -Detaching from program: /opt/vagrant_ruby/bin/ruby, process 1343 -$ diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index b57bc0a1119..0ff1afa86ed 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -8,9 +8,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w This is the GitLab Support Team's collection of information regarding the GitLab Rails console, for use while troubleshooting. It is listed here for transparency, -and it may be useful for users with experience with these tools. If you are currently +and for users with experience with these tools. If you are currently having an issue with GitLab, it is highly recommended that you first check -our guide on [navigating our Rails console](navigating_gitlab_via_rails_console.md), +our guide on [our Rails console](../operations/rails_console.md), and your [support options](https://about.gitlab.com/support/), before attempting to use this information. @@ -80,7 +80,7 @@ Notify.test_email(u.email, "Test email for #{u.name}", 'Test email').deliver_now ## Limiting output -Adding a semicolon(`;`) and a follow-up statement at the end of a statement prevents the default implicit return output. This is useful if you are already explicitly printing details and potentially have a lot of return output: +Adding a semicolon(`;`) and a follow-up statement at the end of a statement prevents the default implicit return output. This can be used if you are already explicitly printing details and potentially have a lot of return output: ```ruby puts ActiveRecord::Base.descendants; :ok @@ -100,9 +100,9 @@ project.id # => 2537 ``` -## Open object in irb +## Open object in `irb` -Sometimes it is easier to navigate through a method if you are within the context of the object. You can shim into the namespace of `Object` to let you open `irb` within the context of any object: +Sometimes it is easier to go through a method if you are in the context of the object. You can shim into the namespace of `Object` to let you open `irb` in the context of any object: ```ruby Object.define_method(:irb) { binding.irb } @@ -311,7 +311,7 @@ end ### Bulk update push rules for _all_ projects -For example, enable **Check whether the commit author is a GitLab user** and **Do not allow users to remove Git tags with `git push`** checkboxes, and create a filter for allowing commits from a specific e-mail domain only: +For example, enable **Check whether the commit author is a GitLab user** and **Do not allow users to remove Git tags with `git push`** checkboxes, and create a filter for allowing commits from a specific email domain only: ``` ruby Project.find_each do |p| @@ -443,7 +443,7 @@ p.create_wiki ### creates the wiki project on the filesystem ## Issue boards -### In case of issue boards not loading properly and it's getting time out. We need to call the Issue Rebalancing service to fix this +### In case of issue boards not loading properly and it's getting time out. Call the Issue Rebalancing service to fix this ```ruby p = Project.find_by_full_path('<username-or-group>/<project-name>') @@ -515,9 +515,9 @@ If this all runs successfully, you see an output like the following before being => nil ``` -The exported project is located within a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. +The exported project is located in a `.tar.gz` file in `/var/opt/gitlab/gitlab-rails/uploads/-/system/import_export_upload/export_file/`. -If this fails, [enable verbose logging](navigating_gitlab_via_rails_console.md#looking-up-database-persisted-objects), +If this fails, [enable verbose logging](../operations/rails_console.md#looking-up-database-persisted-objects), repeat the above procedure after, and report the output to [GitLab Support](https://about.gitlab.com/support/). @@ -611,6 +611,13 @@ user.skip_reconfirmation! ### Disable 2fa for single user +**In GitLab 13.5 and later:** + +Use the code under [Disable 2FA | For a single user](../../security/two_factor_authentication.md#for-a-single-user) so that the target user +is notified that 2FA has been disabled. + +**In GitLab 13.4 and earlier:** + ```ruby user = User.find_by_username('<username>') user.disable_two_factor! @@ -629,7 +636,7 @@ User.billable.count ::HistoricalData.max_historical_user_count(from: 1.year.ago.beginning_of_day, to: Time.current.end_of_day) ``` -Using cURL and jq (up to a max 100, see the [pagination docs](../../api/index.md#pagination)): +Using cURL and jq (up to a max 100, see [Pagination](../../api/index.md#pagination)): ```shell curl --silent --header "Private-Token: ********************" \ @@ -1053,6 +1060,9 @@ License.current.expires_at # Is this a trial license? License.current.trial? + +# License ID for lookup on CustomersDot +License.current.license_id ``` ### Check if a project feature is available on the instance @@ -1101,10 +1111,10 @@ License.select(&TYPE).each(&:destroy!) ### Registry Disk Space Usage by Project -As a GitLab administrator, you may need to reduce disk space consumption. +As a GitLab administrator, you may want to reduce disk space consumption. A common culprit is Docker Registry images that are no longer in use. To find the storage broken down by each project, run the following in the -[GitLab Rails console](../troubleshooting/navigating_gitlab_via_rails_console.md): +[GitLab Rails console](../operations/rails_console.md): ```ruby projects_and_size = [["project_id", "creator_id", "registry_size_bytes", "project path"]] @@ -1135,11 +1145,11 @@ end ### Run the Cleanup policy now -Find this content in the [Container Registry troubleshooting docs](../packages/container_registry.md#run-the-cleanup-policy-now). +Find this content in the [Container Registry troubleshooting documentation](../packages/container_registry.md#run-the-cleanup-policy-now). ## Sidekiq -This content has been moved to the [Troubleshooting Sidekiq docs](sidekiq.md). +This content has been moved to [Troubleshooting Sidekiq](sidekiq.md). ## Redis @@ -1275,11 +1285,16 @@ project = Project.find_by_full_path('<group/project>') Geo::RepositorySyncService.new(project).execute ``` -### Blob types newer than uploads/artifacts/LFS +### Blob types +- `Ci::JobArtifact` +- `Ci::PipelineArtifact` +- `LfsObject` +- `MergeRequestDiff` - `Packages::PackageFile` +- `PagesDeployment` - `Terraform::StateVersion` -- `MergeRequestDiff` +- `Upload` `Packages::PackageFile` is used in the following examples, but things generally work the same for the other Blob types. diff --git a/doc/administration/troubleshooting/group_saml_scim.md b/doc/administration/troubleshooting/group_saml_scim.md index c6a102e87ee..145eb5f65ae 100644 --- a/doc/administration/troubleshooting/group_saml_scim.md +++ b/doc/administration/troubleshooting/group_saml_scim.md @@ -60,6 +60,10 @@ User claims and attributes: IdP links and certificate: +NOTE: +Google Workspace displays a SHA256 fingerprint. To retrieve the SHA1 fingerprint required by GitLab for configuring SAML, download the certificate and calculate the SHA1 certificate +fingerprint. + ![Google Workspace Links and Certificate](img/GoogleWorkspace-linkscert_v14_10.png) ## Okta @@ -109,3 +113,95 @@ Adding a user: SSO settings: ![OneLogin SSO settings](img/OneLogin-SSOsettings.png) + +## SAML response example + +When a user signs in using SAML, GitLab receives a SAML response. The SAML response can be found in `production.log` logs as a base64-encoded message. Locate the response by +searching for `SAMLResponse`. The decoded SAML response is in XML format. For example: + +```xml +<?xml version="1.0" encoding="UTF-8"?> +<saml2p:Response xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol" xmlns:xs="http://www.w3.org/2001/XMLSchema" Destination="https://gitlabexample/-/saml/callback" ID="id4898983630840142426821432" InResponseTo="_c65e4c88-9425-4472-b42c-37f4186ac0ee" IssueInstant="2022-05-30T21:30:35.696Z" Version="2.0"> + <saml2:Issuer xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">http://www.okta.com/exk2y6j57o1Pdr2lI8qh7</saml2:Issuer> + <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"> + <ds:SignedInfo> + <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> + <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/> + <ds:Reference URI="#id4898983630840142426821432"> + <ds:Transforms> + <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/> + <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> + <ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="xs"/> + </ds:Transform> + </ds:Transforms> + <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/> + <ds:DigestValue>neiQvv9d3OgS4GZW8Nptp4JhjpKs3GCefibn+vmRgk4=</ds:DigestValue> + </ds:Reference> + </ds:SignedInfo> + <ds:SignatureValue>dMsQX8ivi...HMuKGhyLRvabGU6CuPrf7==</ds:SignatureValue> + <ds:KeyInfo> + <ds:X509Data> + <ds:X509Certificate>MIIDq...cptGr3vN9TQ==</ds:X509Certificate> + </ds:X509Data> + </ds:KeyInfo> + </ds:Signature> + <saml2p:Status xmlns:saml2p="urn:oasis:names:tc:SAML:2.0:protocol"> + <saml2p:StatusCode Value="urn:oasis:names:tc:SAML:2.0:status:Success"/> + </saml2p:Status> + <saml2:Assertion xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" xmlns:xs="http://www.w3.org/2001/XMLSchema" ID="id489" IssueInstant="2022-05-30T21:30:35.696Z" Version="2.0"> + <saml2:Issuer xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" Format="urn:oasis:names:tc:SAML:2.0:nameid-format:entity">http://www.okta.com/exk2y6j57o1Pdr2lI8qh7</saml2:Issuer> + <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#"> + <ds:SignedInfo> + <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/> + <ds:SignatureMethod Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/> + <ds:Reference URI="#id48989836309833801859473359"> + <ds:Transforms> + <ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/> + <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"> + <ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="xs"/> + </ds:Transform> + </ds:Transforms> + <ds:DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/> + <ds:DigestValue>MaIsoi8hbT9gsi/mNZsz449mUuAcuEWY0q3bc4asOQs=</ds:DigestValue> + </ds:Reference> + </ds:SignedInfo> + <ds:SignatureValue>dMsQX8ivi...HMuKGhyLRvabGU6CuPrf7==<</ds:SignatureValue> + <ds:KeyInfo> + <ds:X509Data> + <ds:X509Certificate>MIIDq...cptGr3vN9TQ==</ds:X509Certificate> + </ds:X509Data> + </ds:KeyInfo> + </ds:Signature> + <saml2:Subject xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"> + <saml2:NameID Format="urn:oasis:names:tc:SAML:2.0:nameid-format:persistent">useremail@domain.com</saml2:NameID> + <saml2:SubjectConfirmation Method="urn:oasis:names:tc:SAML:2.0:cm:bearer"> + <saml2:SubjectConfirmationData InResponseTo="_c65e4c88-9425-4472-b42c-37f4186ac0ee" NotOnOrAfter="2022-05-30T21:35:35.696Z" Recipient="https://gitlab.example.com/-/saml/callback"/> + </saml2:SubjectConfirmation> + </saml2:Subject> + <saml2:Conditions xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" NotBefore="2022-05-30T21:25:35.696Z" NotOnOrAfter="2022-05-30T21:35:35.696Z"> + <saml2:AudienceRestriction> + <saml2:Audience>https://gitlab.example.com/</saml2:Audience> + </saml2:AudienceRestriction> + </saml2:Conditions> + <saml2:AuthnStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion" AuthnInstant="2022-05-30T21:30:35.696Z" SessionIndex="_c65e4c88-9425-4472-b42c-37f4186ac0ee"> + <saml2:AuthnContext> + <saml2:AuthnContextClassRef>urn:oasis:names:tc:SAML:2.0:ac:classes:PasswordProtectedTransport</saml2:AuthnContextClassRef> + </saml2:AuthnContext> + </saml2:AuthnStatement> + <saml2:AttributeStatement xmlns:saml2="urn:oasis:names:tc:SAML:2.0:assertion"> + <saml2:Attribute Name="email" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> + <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">useremail@domain.com</saml2:AttributeValue> + </saml2:Attribute> + <saml2:Attribute Name="firtname" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> + <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">John</saml2:AttributeValue> + </saml2:Attribute> + <saml2:Attribute Name="lastname" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> + <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Doe</saml2:AttributeValue> + </saml2:Attribute> + <saml2:Attribute Name="Groups" NameFormat="urn:oasis:names:tc:SAML:2.0:attrname-format:unspecified"> + <saml2:AttributeValue xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="xs:string">Super-awesome-group</saml2:AttributeValue> + </saml2:Attribute> + </saml2:AttributeStatement> + </saml2:Assertion> +</saml2p:Response> +``` diff --git a/doc/administration/troubleshooting/index.md b/doc/administration/troubleshooting/index.md index 7d40a9e9683..7fe731bda66 100644 --- a/doc/administration/troubleshooting/index.md +++ b/doc/administration/troubleshooting/index.md @@ -17,13 +17,14 @@ installation. - [Sidekiq](sidekiq.md) - [GitLab Rails console cheat sheet](gitlab_rails_cheat_sheet.md) - [Group SAML and SCIM troubleshooting](group_saml_scim.md) **(PREMIUM SAAS)** -- [Kubernetes cheat sheet](kubernetes_cheat_sheet.md) +- [Kubernetes cheat sheet](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html) - [Linux cheat sheet](linux_cheat_sheet.md) - [Parsing GitLab logs with `jq`](log_parsing.md) -- [Navigating GitLab via Rails console](navigating_gitlab_via_rails_console.md) - [Diagnostics tools](diagnostics_tools.md) -- [Debugging tips](debug.md) - [Tracing requests with correlation ID](tracing_correlation_id.md) +Some feature documentation pages also have a troubleshooting section at the end +that you can check for feature-specific help. + If you need a testing environment to troubleshoot, see the [apps for a testing environment](test_environments.md). diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index 0c93d1ab3ee..15ec8d5940b 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -1,295 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference +redirect_to: 'https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html' +remove_date: '2022-10-05' --- -# Kubernetes, GitLab, and you **(FREE SELF)** +This document was moved to [another location](https://docs.gitlab.com/charts/troubleshooting/kubernetes_cheat_sheet.html). -This is a list of useful information regarding Kubernetes that the GitLab Support -Team sometimes uses while troubleshooting. GitLab is making this public, so that anyone -can make use of the Support team's collected knowledge - -WARNING: -These commands **can alter or break** your Kubernetes components so use these at your own risk. - -If you are on a [paid tier](https://about.gitlab.com/pricing/) and are not sure how -to use these commands, it is best to [contact Support](https://about.gitlab.com/support/) -and they will assist you with any issues you are having. - -## Generic Kubernetes commands - -- How to authorize to your GCP project (can be especially useful if you have projects - under different GCP accounts): - - ```shell - gcloud auth login - ``` - -- How to access Kubernetes dashboard: - - ```shell - # for minikube: - minikube dashboard —url - # for non-local installations if access via Kubectl is configured: - kubectl proxy - ``` - -- How to SSH to a Kubernetes node and enter the container as root - <https://github.com/kubernetes/kubernetes/issues/30656>: - - - For GCP, you may find the node name and run `gcloud compute ssh node-name`. - - List containers using `docker ps`. - - Enter container using `docker exec --user root -ti container-id bash`. - -- How to copy a file from local machine to a pod: - - ```shell - kubectl cp file-name pod-name:./destination-path - ``` - -- What to do with pods in `CrashLoopBackoff` status: - - - Check logs via Kubernetes dashboard. - - Check logs via Kubectl: - - ```shell - kubectl logs <webservice pod> -c dependencies - ``` - -- How to tail all Kubernetes cluster events in real time: - - ```shell - kubectl get events -w --all-namespaces - ``` - -- How to get logs of the previously terminated pod instance: - - ```shell - kubectl logs <pod-name> --previous - ``` - - No logs are kept in the containers/pods themselves. Everything is written to `stdout`. - This is the principle of Kubernetes, read [Twelve-factor app](https://12factor.net/) - for details. - -- How to get cron jobs configured on a cluster - - ```shell - kubectl get cronjobs - ``` - - When one configures [cron-based backups](https://docs.gitlab.com/charts/backup-restore/backup.html#cron-based-backup), - you will be able to see the new schedule here. Some details about the schedules can be found - in [Running Automated Tasks with a CronJob](https://kubernetes.io/docs/tasks/job/automated-tasks-with-cron-jobs/#creating-a-cron-job) - -## GitLab-specific Kubernetes information - -- Minimal configuration that can be used to [test a Kubernetes Helm chart](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/620). - -- Tailing logs of a separate pod. An example for a `webservice` pod: - - ```shell - kubectl logs gitlab-webservice-54fbf6698b-hpckq -c webservice - ``` - -- Tail and follow all pods that share a label (in this case, `webservice`): - - ```shell - # all containers in the webservice pods - kubectl logs -f -l app=webservice --all-containers=true --max-log-requests=50 - - # only the webservice containers in all webservice pods - kubectl logs -f -l app=webservice -c webservice --max-log-requests=50 - ``` - -- One can stream logs from all containers at once, similar to the Omnibus - command `gitlab-ctl tail`: - - ```shell - kubectl logs -f -l release=gitlab --all-containers=true --max-log-requests=100 - ``` - -- Check all events in the `gitlab` namespace (the namespace name can be different if you - specified a different one when deploying the Helm chart): - - ```shell - kubectl get events -w --namespace=gitlab - ``` - -- Most of the useful GitLab tools (console, Rake tasks, etc) are found in the toolbox - pod. You may enter it and run commands inside or run them from the outside. - - NOTE: - The `task-runner` pod was renamed to `toolbox` in GitLab 14.2 (charts 5.2). - - ```shell - # find the pod - kubectl --namespace gitlab get pods -lapp=toolbox - - # enter it - kubectl exec -it <toolbox-pod-name> -- bash - - # open rails console - # rails console can be also called from other GitLab pods - /srv/gitlab/bin/rails console - - # source-style commands should also work - cd /srv/gitlab && bundle exec rake gitlab:check RAILS_ENV=production - - # run GitLab check. The output can be confusing and invalid because of the specific structure of GitLab installed via helm chart - /usr/local/bin/gitlab-rake gitlab:check - - # open console without entering pod - kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails console - - # check the status of DB migrations - kubectl exec -it <toolbox-pod-name> -- /usr/local/bin/gitlab-rake db:migrate:status - ``` - - You can also use `gitlab-rake`, instead of `/usr/local/bin/gitlab-rake`. - -- Troubleshooting **Infrastructure > Kubernetes clusters** integration: - - - Check the output of `kubectl get events -w --all-namespaces`. - - Check the logs of pods within `gitlab-managed-apps` namespace. - - On the side of GitLab check Sidekiq log and Kubernetes log. When GitLab is installed - via Helm Chart, `kubernetes.log` can be found inside the Sidekiq pod. - -- How to get your initial administrator password <https://docs.gitlab.com/charts/installation/deployment.html#initial-login>: - - ```shell - # find the name of the secret containing the password - kubectl get secrets | grep initial-root - # decode it - kubectl get secret <secret-name> -ojsonpath={.data.password} | base64 --decode ; echo - ``` - -- How to connect to a GitLab PostgreSQL database. - - NOTE: - The `task-runner` pod was renamed to `toolbox` in GitLab 14.2 (charts 5.2). - - In GitLab 14.2 (chart 5.2) and later: - - ```shell - kubectl exec -it <toolbox-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password --database main - ``` - - In GitLab 14.1 (chart 5.1) and earlier: - - ```shell - kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password - ``` - -- How to get information about Helm installation status: - - ```shell - helm status name-of-installation - ``` - -- How to update GitLab installed using Helm Chart: - - ```shell - helm repo upgrade - - # get current values and redirect them to yaml file (analogue of gitlab.rb values) - helm get values <release name> > gitlab.yaml - - # run upgrade itself - helm upgrade <release name> <chart path> -f gitlab.yaml - ``` - - After <https://gitlab.com/gitlab-org/charts/gitlab/-/issues/780> is fixed, it should - be possible to use [Updating GitLab using the Helm Chart](https://docs.gitlab.com/charts/index.html#updating-gitlab-using-the-helm-chart) - for upgrades. - -- How to apply changes to GitLab configuration: - - - Modify the `gitlab.yaml` file. - - Run the following command to apply changes: - - ```shell - helm upgrade <release name> <chart path> -f gitlab.yaml - ``` - -- How to get the manifest for a release. It can be useful because it contains the information about -all Kubernetes resources and dependent charts: - - ```shell - helm get manifest <release name> - ``` - -## Installation of minimal GitLab configuration via minikube on macOS - -This section is based on [Developing for Kubernetes with minikube](https://docs.gitlab.com/charts/development/minikube/index.html) -and [Helm](https://docs.gitlab.com/charts/installation/tools.html#helm). Refer -to those documents for details. - -- Install Kubectl via Homebrew: - - ```shell - brew install kubernetes-cli - ``` - -- Install minikube via Homebrew: - - ```shell - brew cask install minikube - ``` - -- Start minikube and configure it. If minikube cannot start, try running `minikube delete && minikube start` - and repeat the steps: - - ```shell - minikube start --cpus 3 --memory 8192 # minimum amount for GitLab to work - minikube addons enable ingress - ``` - -- Install Helm via Homebrew and initialize it: - - ```shell - brew install helm - ``` - -- Copy the [minikube minimum values YAML file](https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml) - to your workstation: - - ```shell - curl --output values.yaml "https://gitlab.com/gitlab-org/charts/gitlab/raw/master/examples/values-minikube-minimum.yaml" - ``` - -- Find the IP address in the output of `minikube ip` and update the YAML file with - this IP address. - -- Install the GitLab Helm Chart: - - ```shell - helm repo add gitlab https://charts.gitlab.io - helm install gitlab -f <path-to-yaml-file> gitlab/gitlab - ``` - - If you want to modify some GitLab settings, you can use the above-mentioned configuration - as a base and create your own YAML file. - -- Monitor the installation progress via `helm status gitlab` and `minikube dashboard`. - The installation could take up to 20-30 minutes depending on the amount of resources - on your workstation. - -- When all the pods show either a `Running` or `Completed` status, get the GitLab password as - described in [Initial login](https://docs.gitlab.com/charts/installation/deployment.html#initial-login), - and log in to GitLab via the UI. It will be accessible via `https://gitlab.domain` - where `domain` is the value provided in the YAML file. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after 2022-10-05. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/linux_cheat_sheet.md b/doc/administration/troubleshooting/linux_cheat_sheet.md index 0245af39e45..66d5fb82936 100644 --- a/doc/administration/troubleshooting/linux_cheat_sheet.md +++ b/doc/administration/troubleshooting/linux_cheat_sheet.md @@ -9,7 +9,7 @@ type: reference This is the GitLab Support Team's collection of information regarding Linux, that they sometimes use while troubleshooting. It is listed here for transparency, -and it may be useful for users with experience with Linux. If you are currently +and for users with experience with Linux. If you are currently having an issue with GitLab, you may want to check your [support options](https://about.gitlab.com/support/) first, before attempting to use this information. @@ -107,6 +107,9 @@ grep -2 search_term <filename> # Search on all files in directory (recursively) grep -r search_term <directory> +# Grep namespace/project/name of a GitLab repository +grep 'fullpath' /var/opt/gitlab/git-data/repositories/@hashed/<repo hash>/.git/config + # search through *.gz files is the same except with zgrep zgrep search_term <filename> @@ -126,6 +129,7 @@ history # Search through command history <ctrl>-R + # Execute last command with sudo sudo !! ``` diff --git a/doc/administration/troubleshooting/log_parsing.md b/doc/administration/troubleshooting/log_parsing.md index 4cc62c08f4f..0320b2e52ce 100644 --- a/doc/administration/troubleshooting/log_parsing.md +++ b/doc/administration/troubleshooting/log_parsing.md @@ -47,7 +47,7 @@ grep <TERM> <FILE> | jq . jq -cR 'fromjson?' file.json | jq <COMMAND> ``` -By default `jq` will error out when it encounters a line that is not valid JSON. +By default `jq` errors out when it encounters a line that is not valid JSON. This skips over all invalid lines and parses the rest. #### Print a JSON log's time range @@ -152,6 +152,51 @@ CT: 297 ROUTE: /api/:version/projects/:id/repository/tags DURS: 731.39, CT: 190 ROUTE: /api/:version/projects/:id/repository/commits DURS: 1079.02, 979.68, 958.21 ``` +### Print top API user agents + +```shell +jq --raw-output '[.route, .ua] | @tsv' api_json.log | sort | uniq -c | sort -n +``` + +**Example output**: + +```plaintext + 89 /api/:version/usage_data/increment_unique_users # plus browser details + 567 /api/:version/jobs/:id/trace gitlab-runner # plus version details +1234 /api/:version/internal/allowed GitLab-Shell +``` + +This sample response seems normal. A custom tool or script might be causing a high load +if the output contains many: + +- Third party libraries like `python-requests` or `curl`. +- [GitLab CLI clients](https://about.gitlab.com/partners/technology-partners/#cli-clients). + +You can also [use `fast-stats top`](#parsing-gitlab-logs-with-jq) to extract performance statistics. + +### Parsing `gitlab-workhorse/current` + +### Print top Workhorse user agents + +```shell +jq --raw-output '[.uri, .user_agent] | @tsv' current | sort | uniq -c | sort -n +``` + +**Example output**: + +```plaintext + 89 /api/graphql # plus browser details + 567 /api/v4/internal/allowed GitLab-Shell +1234 /api/v4/jobs/request gitlab-runner # plus version details +``` + +Similar to the [API `ua` data](#print-top-api-user-agents), +deviations from this common order might indicate scripts that could be optimized. + +The performance impact of runners checking for new jobs can be reduced by increasing +[the `check_interval` setting](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section), +for example. + ### Parsing `gitlab-rails/geo.log` #### Find most common Geo sync errors @@ -166,7 +211,7 @@ jq --raw-output 'select(.severity == "ERROR") | [.project_path, .message] | @tsv ### Parsing `gitaly/current` -The following examples are useful to [troubleshoot Gitaly](../gitaly/troubleshooting.md). +Use the following examples to [troubleshoot Gitaly](../gitaly/troubleshooting.md). #### Find all Gitaly requests sent from web UI diff --git a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md index 51ef3d95a4e..09a5cb8d185 100644 --- a/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md +++ b/doc/administration/troubleshooting/navigating_gitlab_via_rails_console.md @@ -1,465 +1,11 @@ --- -stage: Systems -group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../operations/rails_console.md' +remove_date: '2022-10-05' --- -# Navigating GitLab via Rails console **(FREE SELF)** +This document was moved to [another location](../operations/rails_console.md). -At the heart of GitLab is a web application [built using the Ruby on Rails -framework](https://about.gitlab.com/blog/2018/10/29/why-we-use-rails-to-build-gitlab/). -Thanks to this, we also get access to the amazing tools built right into Rails. -This guide introduces the [Rails console](../operations/rails_console.md#starting-a-rails-console-session) -and the basics of interacting with your GitLab instance from the command line. - -WARNING: -The Rails console interacts directly with your GitLab instance. In many cases, -there are no handrails to prevent you from permanently modifying, corrupting -or destroying production data. If you would like to explore the Rails console -with no consequences, you are strongly advised to do so in a test environment. - -This guide is targeted at GitLab system administrators who are troubleshooting -a problem or must retrieve some data that can only be done through direct -access of the GitLab application. Basic knowledge of Ruby is needed (try [this -30-minute tutorial](https://try.ruby-lang.org/) for a quick introduction). -Rails experience is helpful to have but not a must. - -## Starting a Rails console session - -Your type of GitLab installation determines how -[to start a rails console](../operations/rails_console.md). - -The following code examples take place inside the Rails console and also -assume an Omnibus GitLab installation. - -## Active Record objects - -### Looking up database-persisted objects - -Under the hood, Rails uses [Active Record](https://guides.rubyonrails.org/active_record_basics.html), -an object-relational mapping system, to read, write, and map application objects -to the PostgreSQL database. These mappings are handled by Active Record models, -which are Ruby classes defined in a Rails app. For GitLab, the model classes -can be found at `/opt/gitlab/embedded/service/gitlab-rails/app/models`. - -Let's enable debug logging for Active Record so we can see the underlying -database queries made: - -```ruby -ActiveRecord::Base.logger = Logger.new($stdout) -``` - -Now, let's try retrieving a user from the database: - -```ruby -user = User.find(1) -``` - -Which would return: - -```ruby -D, [2020-03-05T16:46:25.571238 #910] DEBUG -- : User Load (1.8ms) SELECT "users".* FROM "users" WHERE "users"."id" = 1 LIMIT 1 -=> #<User id:1 @root> -``` - -We can see that we've queried the `users` table in the database for a row whose -`id` column has the value `1`, and Active Record has translated that database -record into a Ruby object that we can interact with. Try some of the following: - -- `user.username` -- `user.created_at` -- `user.admin` - -By convention, column names are directly translated into Ruby object attributes, -so you should be able to do `user.<column_name>` to view the attribute's value. - -Also by convention, Active Record class names (singular and in camel case) map -directly onto table names (plural and in snake case) and vice versa. For example, -the `users` table maps to the `User` class, while the `application_settings` -table maps to the `ApplicationSetting` class. - -You can find a list of tables and column names in the Rails database schema, -available at `/opt/gitlab/embedded/service/gitlab-rails/db/schema.rb`. - -You can also look up an object from the database by attribute name: - -```ruby -user = User.find_by(username: 'root') -``` - -Which would return: - -```ruby -D, [2020-03-05T17:03:24.696493 #910] DEBUG -- : User Load (2.1ms) SELECT "users".* FROM "users" WHERE "users"."username" = 'root' LIMIT 1 -=> #<User id:1 @root> -``` - -Give the following a try: - -- `User.find_by(email: 'admin@example.com')` -- `User.where.not(admin: true)` -- `User.where('created_at < ?', 7.days.ago)` - -Did you notice that the last two commands returned an `ActiveRecord::Relation` -object that appeared to contain multiple `User` objects? - -Up to now, we've been using `.find` or `.find_by`, which are designed to return -only a single object (notice the `LIMIT 1` in the generated SQL query?). -`.where` is used when it is desirable to get a collection of objects. - -Let's get a collection of non-administrator users and see what we can do with it: - -```ruby -users = User.where.not(admin: true) -``` - -Which would return: - -```ruby -D, [2020-03-05T17:11:16.845387 #910] DEBUG -- : User Load (2.8ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE LIMIT 11 -=> #<ActiveRecord::Relation [#<User id:3 @support-bot>, #<User id:7 @alert-bot>, #<User id:5 @carrie>, #<User id:4 @bernice>, #<User id:2 @anne>]> -``` - -Now, try the following: - -- `users.count` -- `users.order(created_at: :desc)` -- `users.where(username: 'support-bot')` - -In the last command, we see that we can chain `.where` statements to generate -more complex queries. Notice also that while the collection returned contains -only a single object, we cannot directly interact with it: - -```ruby -users.where(username: 'support-bot').username -``` - -Which would return: - -```ruby -Traceback (most recent call last): - 1: from (irb):37 -D, [2020-03-05T17:18:25.637607 #910] DEBUG -- : User Load (1.6ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE AND "users"."username" = 'support-bot' LIMIT 11 -NoMethodError (undefined method `username' for #<ActiveRecord::Relation [#<User id:3 @support-bot>]>) -Did you mean? by_username -``` - -Let's retrieve the single object from the collection by using the `.first` -method to get the first item in the collection: - -```ruby -users.where(username: 'support-bot').first.username -``` - -We now get the result we wanted: - -```ruby -D, [2020-03-05T17:18:30.406047 #910] DEBUG -- : User Load (2.6ms) SELECT "users".* FROM "users" WHERE "users"."admin" != TRUE AND "users"."username" = 'support-bot' ORDER BY "users"."id" ASC LIMIT 1 -=> "support-bot" -``` - -For more on different ways to retrieve data from the database using Active -Record, please see the [Active Record Query Interface documentation](https://guides.rubyonrails.org/active_record_querying.html). - -### Modifying Active Record objects - -In the previous section, we learned about retrieving database records using -Active Record. Now, let's learn how to write changes to the database. - -First, let's retrieve the `root` user: - -```ruby -user = User.find_by(username: 'root') -``` - -Next, let's try updating the user's password: - -```ruby -user.password = 'password' -user.save -``` - -Which would return: - -```ruby -Enqueued ActionMailer::MailDeliveryJob (Job ID: 05915c4e-c849-4e14-80bb-696d5ae22065) to Sidekiq(mailers) with arguments: "DeviseMailer", "password_change", "deliver_now", #<GlobalID:0x00007f42d8ccebe8 @uri=#<URI::GID gid://gitlab/User/1>> -=> true -``` - -Here, we see that the `.save` command returned `true`, indicating that the -password change was successfully saved to the database. - -We also see that the save operation triggered some other action -- in this case -a background job to deliver an email notification. This is an example of an -[Active Record callback](https://guides.rubyonrails.org/active_record_callbacks.html) --- code which is designated to run in response to events in the Active Record -object life cycle. This is also why using the Rails console is preferred when -direct changes to data is necessary as changes made via direct database queries -do not trigger these callbacks. - -It's also possible to update attributes in a single line: - -```ruby -user.update(password: 'password') -``` - -Or update multiple attributes at once: - -```ruby -user.update(password: 'password', email: 'hunter2@example.com') -``` - -Now, let's try something different: - -```ruby -# Retrieve the object again so we get its latest state -user = User.find_by(username: 'root') -user.password = 'password' -user.password_confirmation = 'hunter2' -user.save -``` - -This returns `false`, indicating that the changes we made were not saved to the -database. You can probably guess why, but let's find out for sure: - -```ruby -user.save! -``` - -This should return: - -```ruby -Traceback (most recent call last): - 1: from (irb):64 -ActiveRecord::RecordInvalid (Validation failed: Password confirmation doesn't match Password) -``` - -Aha! We've tripped an [Active Record Validation](https://guides.rubyonrails.org/active_record_validations.html). -Validations are business logic put in place at the application-level to prevent -unwanted data from being saved to the database and in most cases come with -helpful messages letting you know how to fix the problem inputs. - -We can also add the bang (Ruby speak for `!`) to `.update`: - -```ruby -user.update!(password: 'password', password_confirmation: 'hunter2') -``` - -In Ruby, method names ending with `!` are commonly known as "bang methods". By -convention, the bang indicates that the method directly modifies the object it -is acting on, as opposed to returning the transformed result and leaving the -underlying object untouched. For Active Record methods that write to the -database, bang methods also serve an additional function: they raise an -explicit exception whenever an error occurs, instead of just returning `false`. - -We can also skip validations entirely: - -```ruby -# Retrieve the object again so we get its latest state -user = User.find_by(username: 'root') -user.password = 'password' -user.password_confirmation = 'hunter2' -user.save!(validate: false) -``` - -This is not recommended, as validations are usually put in place to ensure the -integrity and consistency of user-provided data. - -A validation error prevents the entire object from being saved to -the database. You can see a little of this in the section below. If you're getting -a mysterious red banner in the GitLab UI when submitting a form, this can often -be the fastest way to get to the root of the problem. - -### Interacting with Active Record objects - -At the end of the day, Active Record objects are just normal Ruby objects. As -such, we can define methods on them which perform arbitrary actions. - -For example, GitLab developers have added some methods which help with -two-factor authentication: - -```ruby -def disable_two_factor! - transaction do - update( - otp_required_for_login: false, - encrypted_otp_secret: nil, - encrypted_otp_secret_iv: nil, - encrypted_otp_secret_salt: nil, - otp_grace_period_started_at: nil, - otp_backup_codes: nil - ) - self.u2f_registrations.destroy_all # rubocop: disable DestroyAll - end -end - -def two_factor_enabled? - two_factor_otp_enabled? || two_factor_u2f_enabled? -end -``` - -(See: `/opt/gitlab/embedded/service/gitlab-rails/app/models/user.rb`) - -We can then use these methods on any user object: - -```ruby -user = User.find_by(username: 'root') -user.two_factor_enabled? -user.disable_two_factor! -``` - -Some methods are defined by gems, or Ruby software packages, which GitLab uses. -For example, the [StateMachines](https://github.com/state-machines/state_machines-activerecord) -gem which GitLab uses to manage user state: - -```ruby -state_machine :state, initial: :active do - event :block do - - ... - - event :activate do - - ... - -end -``` - -Give it a try: - -```ruby -user = User.find_by(username: 'root') -user.state -user.block -user.state -user.activate -user.state -``` - -Earlier, we mentioned that a validation error prevents the entire object -from being saved to the database. Let's see how this can have unexpected -interactions: - -```ruby -user.password = 'password' -user.password_confirmation = 'hunter2' -user.block -``` - -We get `false` returned! Let's find out what happened by adding a bang as we did -earlier: - -```ruby -user.block! -``` - -Which would return: - -```ruby -Traceback (most recent call last): - 1: from (irb):87 -StateMachines::InvalidTransition (Cannot transition state via :block from :active (Reason(s): Password confirmation doesn't match Password)) -``` - -We see that a validation error from what feels like a completely separate -attribute comes back to haunt us when we try to update the user in any way. - -In practical terms, we sometimes see this happen with GitLab administration settings -- -validations are sometimes added or changed in a GitLab update, resulting in -previously saved settings now failing validation. Because you can only update -a subset of settings at once through the UI, in this case the only way to get -back to a good state is direct manipulation via Rails console. - -### Commonly used Active Record models and how to look up objects - -**Get a user by primary email address or username:** - -```ruby -User.find_by(email: 'admin@example.com') -User.find_by(username: 'root') -``` - -**Get a user by primary OR secondary email address:** - -```ruby -User.find_by_any_email('user@example.com') -``` - -The `find_by_any_email` method is a custom method added by GitLab developers rather -than a Rails-provided default method. - -**Get a collection of administrator users:** - -```ruby -User.admins -``` - -`admins` is a [scope convenience method](https://guides.rubyonrails.org/active_record_querying.html#scopes) -which does `where(admin: true)` under the hood. - -**Get a project by its path:** - -```ruby -Project.find_by_full_path('group/subgroup/project') -``` - -`find_by_full_path` is a custom method added by GitLab developers rather -than a Rails-provided default method. - -**Get a project's issue or merge request by its numeric ID:** - -```ruby -project = Project.find_by_full_path('group/subgroup/project') -project.issues.find_by(iid: 42) -project.merge_requests.find_by(iid: 42) -``` - -`iid` means "internal ID" and is how we keep issue and merge request IDs -scoped to each GitLab project. - -**Get a group by its path:** - -```ruby -Group.find_by_full_path('group/subgroup') -``` - -**Get a group's related groups:** - -```ruby -group = Group.find_by_full_path('group/subgroup') - -# Get a group's parent group -group.parent - -# Get a group's child groups -group.children -``` - -**Get a group's projects:** - -```ruby -group = Group.find_by_full_path('group/subgroup') - -# Get group's immediate child projects -group.projects - -# Get group's child projects, including those in subgroups -group.all_projects -``` - -**Get CI pipeline or builds:** - -```ruby -Ci::Pipeline.find(4151) -Ci::Build.find(66124) -``` - -The pipeline and job ID numbers increment globally across your GitLab -instance, so there's no requirement to use an internal ID attribute to look them up, -unlike with issues or merge requests. - -**Get the current application settings object:** - -```ruby -ApplicationSetting.current -``` +<!-- This redirect file can be deleted after <2022-10-05>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index cdbf786bdb2..61b661d45f8 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -54,7 +54,7 @@ This section is for links to information elsewhere in the GitLab documentation. - Including [troubleshooting](../postgresql/replication_and_failover.md#troubleshooting) `gitlab-ctl patroni check-leader` and PgBouncer errors. -- [Developer database documentation](../../development/index.md#database-guides), +- [Developer database documentation](../../development/feature_development.md#database-guides), some of which is absolutely not for production use. Including: - Understanding EXPLAIN plans. diff --git a/doc/administration/troubleshooting/ssl.md b/doc/administration/troubleshooting/ssl.md index d5d50127ad5..e1cd92a788f 100644 --- a/doc/administration/troubleshooting/ssl.md +++ b/doc/administration/troubleshooting/ssl.md @@ -9,7 +9,7 @@ type: reference This page contains a list of common SSL-related errors and scenarios that you may encounter while working with GitLab. It should serve as an addition to the -main SSL docs available here: +main SSL documentation: - [Omnibus SSL Configuration](https://docs.gitlab.com/omnibus/settings/ssl.html). - [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html). @@ -110,8 +110,7 @@ https://gitlab.domain.tld/api/v4/jobs/request: Post https://gitlab.domain.tld/ap x509: certificate signed by unknown authority ``` -If you encounter a similar problem, add your certificate to `/etc/gitlab-runner/certs`, -and the restart the runner by running `gitlab-runner restart`. +Follow the details in [Self-signed certificates or custom Certification Authorities for GitLab Runner](https://docs.gitlab.com/runner/configuration/tls-self-signed.html). ## Mirroring a remote GitLab repository that uses a self-signed SSL certificate @@ -237,7 +236,7 @@ remote server that is serving only HTTP. One scenario is that you're using [object storage](../object_storage.md), which isn't served under HTTPS. GitLab is misconfigured and attempts a TLS handshake, -but the object storage will respond with plain HTTP. +but the object storage responds with plain HTTP. ## `schannel: SEC_E_UNTRUSTED_ROOT` @@ -247,7 +246,7 @@ If you're on Windows and get the following error: Fatal: unable to access 'https://gitlab.domain.tld/group/project.git': schannel: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted." ``` -You may need to specify that Git should use OpenSSL: +You must specify that Git should use OpenSSL: ```shell git config --system http.sslbackend openssl diff --git a/doc/administration/troubleshooting/test_environments.md b/doc/administration/troubleshooting/test_environments.md index 94d17ba714e..d240103a51c 100644 --- a/doc/administration/troubleshooting/test_environments.md +++ b/doc/administration/troubleshooting/test_environments.md @@ -45,7 +45,7 @@ docker run --name gitlab_saml -p 8080:8080 -p 8443:8443 \ -d jamedjo/test-saml-idp ``` -The following will also need to go in your `/etc/gitlab/gitlab.rb`. See [our SAML docs](../../integration/saml.md) +The following must also go in your `/etc/gitlab/gitlab.rb`. See [our SAML docs](../../integration/saml.md) for more, as well as the list of [default usernames, passwords, and emails](https://hub.docker.com/r/jamedjo/test-saml-idp/#usage). ```ruby diff --git a/doc/administration/troubleshooting/tracing_correlation_id.md b/doc/administration/troubleshooting/tracing_correlation_id.md index fed3604057b..418dd729066 100644 --- a/doc/administration/troubleshooting/tracing_correlation_id.md +++ b/doc/administration/troubleshooting/tracing_correlation_id.md @@ -32,7 +32,7 @@ documentation for some popular browsers. To locate a relevant request and view its correlation ID: -1. Enable persistent logging in your network monitor. Some actions in GitLab will redirect you quickly after you submit a form, so this will help capture all relevant activity. +1. Enable persistent logging in your network monitor. Some actions in GitLab redirect you quickly after you submit a form, so this helps capture all relevant activity. 1. To help isolate the requests you are looking for, you can filter for `document` requests. 1. Select the request of interest to view further detail. 1. Go to the **Headers** section and look for **Response Headers**. There you should find an `x-request-id` header with a @@ -121,7 +121,7 @@ find /var/log/gitlab -type f -mtime 0 -exec grep 'LOt9hgi1TV4' '{}' '+' ### Searching in distributed architectures If you have done some horizontal scaling in your GitLab infrastructure, then -you will need to search across _all_ of your GitLab nodes. You can do this with +you must search across _all_ of your GitLab nodes. You can do this with some sort of log aggregation software like Loki, ELK, Splunk, or others. You can use a tool like Ansible or PSSH (parallel SSH) that can execute identical commands across your servers in diff --git a/doc/administration/uploads.md b/doc/administration/uploads.md index e618c4787ab..0bd46193d10 100644 --- a/doc/administration/uploads.md +++ b/doc/administration/uploads.md @@ -66,7 +66,7 @@ For source installations the following settings are nested under `uploads:` and | Setting | Description | Default | |---------|-------------|---------| | `enabled` | Enable/disable object storage | `false` | -| `remote_directory` | The bucket name where Uploads will be stored| | +| `remote_directory` | The bucket name where Uploads are stored| | | `proxy_download` | Set to `true` to enable proxying all files served. Option allows to reduce egress traffic as this allows clients to download directly from remote storage instead of proxying all data | `false` | | `connection` | Various connection options described below | | diff --git a/doc/administration/whats-new.md b/doc/administration/whats-new.md index beb47254573..05c769cf20c 100644 --- a/doc/administration/whats-new.md +++ b/doc/administration/whats-new.md @@ -1,7 +1,7 @@ --- -stage: Growth -group: Adoption -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +stage: none +group: unassigned +info: For assistance with this What's new page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-other-projects-and-subjects. --- # What's new **(FREE)** diff --git a/doc/api/access_requests.md b/doc/api/access_requests.md index 32411a2f557..adaae78f1b5 100644 --- a/doc/api/access_requests.md +++ b/doc/api/access_requests.md @@ -17,7 +17,7 @@ following levels are recognized: - Reporter (`20`) - Developer (`30`) - Maintainer (`40`) -- Owner (`50`) - Only valid to set for groups +- Owner (`50`). Valid for projects in [GitLab 14.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/21432). ## List access requests for a group or project diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index 9408e7c25a6..aa942582e9c 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -24,7 +24,7 @@ The following API resources are available in the project context: | Resource | Available endpoints | |:------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | [Access requests](access_requests.md) | `/projects/:id/access_requests` (also available for groups) | -| [Access tokens](project_access_tokens.md) | `/projects/:id/access_tokens` (also available for groups) | +| [Access tokens](project_access_tokens.md) | `/projects/:id/access_tokens` (also available for groups) | | [Agents](cluster_agents.md) | `/projects/:id/cluster_agents` | | [Award emoji](award_emoji.md) | `/projects/:id/issues/.../award_emoji`, `/projects/:id/merge_requests/.../award_emoji`, `/projects/:id/snippets/.../award_emoji` | | [Branches](branches.md) | `/projects/:id/repository/branches/`, `/projects/:id/repository/merged_branches` | @@ -57,6 +57,7 @@ The following API resources are available in the project context: | [Merge request approvals](merge_request_approvals.md) **(PREMIUM)** | `/projects/:id/approvals`, `/projects/:id/merge_requests/.../approvals` | | [Merge requests](merge_requests.md) | `/projects/:id/merge_requests` (also available for groups and standalone) | | [Merge trains](merge_trains.md) | `/projects/:id/merge_trains` | +| [Metadata](metadata.md) | `/metadata` | | [Notes](notes.md) (comments) | `/projects/:id/issues/.../notes`, `/projects/:id/snippets/.../notes`, `/projects/:id/merge_requests/.../notes` (also available for groups) | | [Notification settings](notification_settings.md) | `/projects/:id/notification_settings` (also available for groups and standalone) | | [Packages](packages.md) | `/projects/:id/packages` | @@ -70,7 +71,7 @@ The following API resources are available in the project context: | [Project milestones](milestones.md) | `/projects/:id/milestones` | | [Project snippets](project_snippets.md) | `/projects/:id/snippets` | | [Project templates](project_templates.md) | `/projects/:id/templates` | -| [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/vulnerabilities` | +| [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/vulnerabilities` | | [Project wikis](wikis.md) | `/projects/:id/wikis` | | [Project-level variables](project_level_variables.md) | `/projects/:id/variables` | | [Projects](projects.md) including setting Webhooks | `/projects`, `/projects/:id/hooks` (also available for users) | diff --git a/doc/api/audit_events.md b/doc/api/audit_events.md index 4ddd851ebda..753e01a15aa 100644 --- a/doc/api/audit_events.md +++ b/doc/api/audit_events.md @@ -131,7 +131,8 @@ Example response: ## Group Audit Events -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34078) in GitLab 12.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34078) in GitLab 12.5. +> - [Support for keyset pagination added](https://gitlab.com/gitlab-org/gitlab/-/issues/333968) in GitLab 15.2. The Group Audit Events API allows you to retrieve [group audit events](../administration/audit_events.md#group-events). This API cannot retrieve project audit events. @@ -139,6 +140,10 @@ This API cannot retrieve project audit events. A user with a Owner role (or above) can retrieve group audit events of all users. A user with a Developer or Maintainer role is limited to group audit events based on their individual actions. +This endpoint optionally supports [keyset pagination](index.md#keyset-based-pagination): + +- When requesting consecutive pages of results, we recommend you use keyset pagination. + ### Retrieve all group audit events ```plaintext diff --git a/doc/api/award_emoji.md b/doc/api/award_emoji.md index 61f84dfb812..5b350dd88c6 100644 --- a/doc/api/award_emoji.md +++ b/doc/api/award_emoji.md @@ -23,7 +23,10 @@ See [Award Emoji on Comments](#award-emoji-on-comments) for information on using ### List an awardable's award emojis -Get a list of all award emojis for a specified awardable. +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/335068) in GitLab 15.1 to allow unauthenticated access to public awardables. + +Get a list of all award emojis for a specified awardable. This endpoint can +be accessed without authentication if the awardable is publicly accessible. ```plaintext GET /projects/:id/issues/:issue_iid/award_emoji @@ -85,7 +88,10 @@ Example response: ### Get single award emoji -Get a single award emoji from an issue, snippet, or merge request. +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/335068) in GitLab 15.1 to allow unauthenticated access to public awardables. + +Get a single award emoji from an issue, snippet, or merge request. This endpoint can +be accessed without authentication if the awardable is publicly accessible. ```plaintext GET /projects/:id/issues/:issue_iid/award_emoji/:award_id @@ -206,7 +212,10 @@ adapted to comments on merge requests and snippets. Therefore, you have to repla ### List a comment's award emojis -Get all award emojis for a comment (note). +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/335068) in GitLab 15.1 to allow unauthenticated access to public comments. + +Get all award emojis for a comment (note). This endpoint can +be accessed without authentication if the comment is publicly accessible. ```plaintext GET /projects/:id/issues/:issue_iid/notes/:note_id/award_emoji @@ -251,7 +260,10 @@ Example response: ### Get an award emoji for a comment -Get a single award emoji for a comment (note). +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/335068) in GitLab 15.1 to allow unauthenticated access to public comments. + +Get a single award emoji for a comment (note). This endpoint can +be accessed without authentication if the comment is publicly accessible. ```plaintext GET /projects/:id/issues/:issue_iid/notes/:note_id/award_emoji/:award_id diff --git a/doc/api/broadcast_messages.md b/doc/api/broadcast_messages.md index 2252568be61..6c95bf2fda5 100644 --- a/doc/api/broadcast_messages.md +++ b/doc/api/broadcast_messages.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Activation +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/deploy_keys.md b/doc/api/deploy_keys.md index adeda014af0..40641c6e2f7 100644 --- a/doc/api/deploy_keys.md +++ b/doc/api/deploy_keys.md @@ -6,6 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Deploy keys API **(FREE)** +The deploy keys API can return in responses fingerprints of the public key in the following fields: + +- `fingerprint` (MD5 hash). Not available on FIPS-enabled systems. +- `fingerprint_sha256` (SHA256 hash). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91302) in GitLab 15.2. + ## List all deploy keys **(FREE SELF)** Get a list of all deploy keys across all projects of the GitLab instance. This @@ -34,8 +39,9 @@ Example response: { "id": 1, "title": "Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", - "fingerprint": "7f:72:08:7d:0e:47:48:ec:37:79:b2:76:68:b5:87:65", + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDNJAkI3Wdf0r13c8a5pEExB2YowPWCSVzfZV22pNBc1CuEbyYLHpUyaD0GwpGvFdx2aP7lMEk35k6Rz3ccBF6jRaVJyhsn5VNnW92PMpBJ/P1UebhXwsFHdQf5rTt082cSxWuk61kGWRQtk4ozt/J2DF/dIUVaLvc+z4HomT41fQ==", + "fingerprint": "4a:9d:64:15:ed:3a:e6:07:6e:89:36:b3:3b:03:05:d9", + "fingerprint_sha256": "SHA256:Jrs3LD1Ji30xNLtTVf9NDCj7kkBgPBb2pjvTZ3HfIgU", "created_at": "2013-10-02T10:12:29Z", "projects_with_write_access": [ { @@ -61,8 +67,9 @@ Example response: { "id": 3, "title": "Another Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", - "fingerprint": "64:d3:73:d4:83:70:ab:41:96:68:d5:3d:a5:b0:34:ea", + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDIJFwIL6YNcCgVBLTHgM6hzmoL5vf0ThDKQMWT3HrwCjUCGPwR63vBwn6+/Gx+kx+VTo9FuojzR0O4XfwD3LrYA+oT3ETbn9U4e/VS4AH/G4SDMzgSLwu0YuPe517FfGWhWGQhjiXphkaQ+6bXPmcASWb0RCO5+pYlGIfxv4eFGQ==" + "fingerprint": "0b:cf:58:40:b9:23:96:c7:ba:44:df:0e:9e:87:5e:75", + "fingerprint_sha256": "SHA256:lGI/Ys/Wx7PfMhUO1iuBH92JQKYN+3mhJZvWO4Q5ims", "created_at": "2013-10-02T11:12:29Z", "projects_with_write_access": [] } @@ -92,14 +99,18 @@ Example response: { "id": 1, "title": "Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDNJAkI3Wdf0r13c8a5pEExB2YowPWCSVzfZV22pNBc1CuEbyYLHpUyaD0GwpGvFdx2aP7lMEk35k6Rz3ccBF6jRaVJyhsn5VNnW92PMpBJ/P1UebhXwsFHdQf5rTt082cSxWuk61kGWRQtk4ozt/J2DF/dIUVaLvc+z4HomT41fQ==", + "fingerprint": "4a:9d:64:15:ed:3a:e6:07:6e:89:36:b3:3b:03:05:d9", + "fingerprint_sha256": "SHA256:Jrs3LD1Ji30xNLtTVf9NDCj7kkBgPBb2pjvTZ3HfIgU", "created_at": "2013-10-02T10:12:29Z", "can_push": false }, { "id": 3, "title": "Another Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDIJFwIL6YNcCgVBLTHgM6hzmoL5vf0ThDKQMWT3HrwCjUCGPwR63vBwn6+/Gx+kx+VTo9FuojzR0O4XfwD3LrYA+oT3ETbn9U4e/VS4AH/G4SDMzgSLwu0YuPe517FfGWhWGQhjiXphkaQ+6bXPmcASWb0RCO5+pYlGIfxv4eFGQ==" + "fingerprint": "0b:cf:58:40:b9:23:96:c7:ba:44:df:0e:9e:87:5e:75", + "fingerprint_sha256": "SHA256:lGI/Ys/Wx7PfMhUO1iuBH92JQKYN+3mhJZvWO4Q5ims", "created_at": "2013-10-02T11:12:29Z", "can_push": false } @@ -129,16 +140,18 @@ Parameters: "title": "Key A", "created_at": "2022-05-30T12:28:27.855Z", "expires_at": null, - "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTEVaAtU5wiVducsOa01InRFf7QSTxoAm6Xy0PGv/k48M6xCALa9nY+BzlOv47jUT57 Key", - "fingerprint": "60:8e:10:f0:6a:82:c8:29:5f:bf:c0:38:72:00:6f:8f" + "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILkYXU2fVeO4/0rDCSsswP5iIX2+B6tv15YT3KObgyDl Key", + "fingerprint": "40:8e:fa:df:70:f7:a7:06:1e:0d:6f:ae:f2:27:92:01", + "fingerprint_sha256": "SHA256:Ojq2LZW43BFK/AMP81jBkDGn9YpPWYRNcViKBB44LPU" }, { "id": 2, "title": "Key B", "created_at": "2022-05-30T13:34:56.219Z", "expires_at": null, - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", - "fingerprint": "75:33:44:7e:55:84:dd:70:29:a3:8e:a3:c0:b9:8b:65" + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDNJAkI3Wdf0r13c8a5pEExB2YowPWCSVzfZV22pNBc1CuEbyYLHpUyaD0GwpGvFdx2aP7lMEk35k6Rz3ccBF6jRaVJyhsn5VNnW92PMpBJ/P1UebhXwsFHdQf5rTt082cSxWuk61kGWRQtk4ozt/J2DF/dIUVaLvc+z4HomT41fQ==", + "fingerprint": "4a:9d:64:15:ed:3a:e6:07:6e:89:36:b3:3b:03:05:d9", + "fingerprint_sha256": "SHA256:Jrs3LD1Ji30xNLtTVf9NDCj7kkBgPBb2pjvTZ3HfIgU", } ] ``` @@ -156,8 +169,9 @@ Example response: "title": "Key A", "created_at": "2022-05-30T12:28:27.855Z", "expires_at": "2022-10-30T12:28:27.855Z", - "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTEVaAtU5wiVducsOa01InRFf7QSTxoAm6Xy0PGv/k48M6xCALa9nY+BzlOv47jUT57 Key", - "fingerprint": "60:8e:10:f0:6a:82:c8:29:5f:bf:c0:38:72:00:6f:8f" + "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILkYXU2fVeO4/0rDCSsswP5iIX2+B6tv15YT3KObgyDl Key", + "fingerprint": "40:8e:fa:df:70:f7:a7:06:1e:0d:6f:ae:f2:27:92:01", + "fingerprint_sha256": "SHA256:Ojq2LZW43BFK/AMP81jBkDGn9YpPWYRNcViKBB44LPU" } ] ``` @@ -187,7 +201,9 @@ Example response: { "id": 1, "title": "Public key", - "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", + "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAAgQDNJAkI3Wdf0r13c8a5pEExB2YowPWCSVzfZV22pNBc1CuEbyYLHpUyaD0GwpGvFdx2aP7lMEk35k6Rz3ccBF6jRaVJyhsn5VNnW92PMpBJ/P1UebhXwsFHdQf5rTt082cSxWuk61kGWRQtk4ozt/J2DF/dIUVaLvc+z4HomT41fQ==", + "fingerprint": "4a:9d:64:15:ed:3a:e6:07:6e:89:36:b3:3b:03:05:d9", + "fingerprint_sha256": "SHA256:Jrs3LD1Ji30xNLtTVf9NDCj7kkBgPBb2pjvTZ3HfIgU", "created_at": "2013-10-02T10:12:29Z", "can_push": false } diff --git a/doc/api/deployments.md b/doc/api/deployments.md index fb255bfa226..6831f86e4ea 100644 --- a/doc/api/deployments.md +++ b/doc/api/deployments.md @@ -345,7 +345,7 @@ POST /projects/:id/deployments | `sha` | string | yes | The SHA of the commit that is deployed. | | `ref` | string | yes | The name of the branch or tag that is deployed. | | `tag` | boolean | yes | A boolean that indicates if the deployed ref is a tag (`true`) or not (`false`). | -| `status` | string | no | The status to filter deployments by. One of `created`, `running`, `success`, `failed`, or `canceled`. | +| `status` | string | yes | The status to filter deployments by. One of `running`, `success`, `failed`, or `canceled`. | ```shell curl --data "environment=production&sha=a91957a858320c0e17f3a0eca7cfacbff50ea29a&ref=main&tag=false&status=success" \ @@ -400,7 +400,7 @@ PUT /projects/:id/deployments/:deployment_id |------------------|----------------|----------|---------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user. | | `deployment_id` | integer | yes | The ID of the deployment to update. | -| `status` | string | no | The new status of the deployment. One of `created`, `running`, `success`, `failed`, or `canceled`. | +| `status` | string | yes | The new status of the deployment. One of `running`, `success`, `failed`, or `canceled`. | ```shell curl --request PUT --data "status=success" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/deployments/42" diff --git a/doc/api/dora/metrics.md b/doc/api/dora/metrics.md index f5373d02156..99473ca7b4c 100644 --- a/doc/api/dora/metrics.md +++ b/doc/api/dora/metrics.md @@ -21,14 +21,15 @@ Get project-level DORA metrics. GET /projects/:id/dora/metrics ``` -| Attribute | Type | Required | Description | -|-------------- |-------- |----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../index.md#namespaced-path-encoding) can be accessed by the authenticated user. | -| `metric` | string | yes | The metric name: `deployment_frequency`, `lead_time_for_changes` or `time_to_restore_service`.| -| `start_date` | string | no | Date range to start from. ISO 8601 Date format, for example `2021-03-01`. Default is 3 months ago. | -| `end_date` | string | no | Date range to end at. ISO 8601 Date format, for example `2021-03-01`. Default is the current date. | -| `interval` | string | no | The bucketing interval. One of `all`, `monthly` or `daily`. Default is `daily`. | -| `environment_tier` | string | no | The [tier of the environment](../../ci/environments/index.md#deployment-tier-of-environments). Default is `production`. | +| Attribute | Type | Required | Description | +|:---------------------|:-----------------|:---------|:------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../index.md#namespaced-path-encoding) can be accessed by the authenticated user. | +| `metric` | string | yes | One of `deployment_frequency`, `lead_time_for_changes`, `time_to_restore_service` or `change_failure_rate`. | +| `end_date` | string | no | Date range to end at. ISO 8601 Date format, for example `2021-03-01`. Default is the current date. | +| `environment_tier` | string | no | The [tier of the environment](../../ci/environments/index.md#deployment-tier-of-environments). Default is `production`. Deprecated, please use `environment_tiers`. | +| `environment_tiers` | array of strings | no | The [tiers of the environments](../../ci/environments/index.md#deployment-tier-of-environments). Default is `production`. | +| `interval` | string | no | The bucketing interval. One of `all`, `monthly` or `daily`. Default is `daily`. | +| `start_date` | string | no | Date range to start from. ISO 8601 Date format, for example `2021-03-01`. Default is 3 months ago. | Example request: @@ -61,14 +62,15 @@ Get group-level DORA metrics. GET /groups/:id/dora/metrics ``` -| Attribute | Type | Required | Description | -|-------------- |-------- |----------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../index.md#namespaced-path-encoding) can be accessed by the authenticated user. | -| `metric` | string | yes | One of `deployment_frequency`, `lead_time_for_changes`, `time_to_restore_service` or `change_failure_rate`. | -| `start_date` | string | no | Date range to start from. ISO 8601 Date format, for example `2021-03-01`. Default is 3 months ago. | -| `end_date` | string | no | Date range to end at. ISO 8601 Date format, for example `2021-03-01`. Default is the current date. | -| `interval` | string | no | The bucketing interval. One of `all`, `monthly` or `daily`. Default is `daily`. | -| `environment_tier` | string | no | The [tier of the environment](../../ci/environments/index.md#deployment-tier-of-environments). Default is `production`. | +| Attribute | Type | Required | Description | +|:--------------------|:-----------------|:---------|:------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../index.md#namespaced-path-encoding) can be accessed by the authenticated user. | +| `metric` | string | yes | One of `deployment_frequency`, `lead_time_for_changes`, `time_to_restore_service` or `change_failure_rate`. | +| `end_date` | string | no | Date range to end at. ISO 8601 Date format, for example `2021-03-01`. Default is the current date. | +| `environment_tier` | string | no | The [tier of the environment](../../ci/environments/index.md#deployment-tier-of-environments). Default is `production`. Deprecated, please use `environment_tiers`. | +| `environment_tiers` | array of strings | no | The [tiers of the environments](../../ci/environments/index.md#deployment-tier-of-environments). Default is `production`. | +| `interval` | string | no | The bucketing interval. One of `all`, `monthly` or `daily`. Default is `daily`. | +| `start_date` | string | no | Date range to start from. ISO 8601 Date format, for example `2021-03-01`. Default is 3 months ago. | Example request: @@ -97,9 +99,9 @@ For both the project and group-level endpoints above, the `value` field in the API response has a different meaning depending on the provided `metric` query parameter: -| `metric` query parameter | Description of `value` in response | -| ------------------------ |--------------------------------------------------------------------------------------------------------------------------------------------------------------| -| `deployment_frequency` | The number of successful deployments during the time period. | -| `lead_time_for_changes` | The median number of seconds between the merge of the merge request (MR) and the deployment of the MR's commits for all MRs deployed during the time period. | -| `time_to_restore_service` | The median number of seconds an incident was open during the time period. Available only for production environment. | -| `change_failure_rate` | The number of incidents divided by the number of deployments during the time period. Available only for production environment. | +| `metric` query parameter | Description of `value` in response | +|:---------------------------|:-----------------------------------| +| `change_failure_rate` | The number of incidents divided by the number of deployments during the time period. Available only for production environment. | +| `deployment_frequency` | The number of successful deployments during the time period. | +| `lead_time_for_changes` | The median number of seconds between the merge of the merge request (MR) and the deployment of the MR's commits for all MRs deployed during the time period. | +| `time_to_restore_service` | The median number of seconds an incident was open during the time period. Available only for production environment. | diff --git a/doc/api/dora4_project_analytics.md b/doc/api/dora4_project_analytics.md deleted file mode 100644 index 19c7afd9d22..00000000000 --- a/doc/api/dora4_project_analytics.md +++ /dev/null @@ -1,13 +0,0 @@ ---- -stage: Manage -group: Optimize -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, api -remove_date: '2022-05-18' -redirect_to: 'dora/metrics.md' ---- - -# DORA4 Analytics Project API (removed) **(ULTIMATE)** - -WARNING: -This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/323713) in 13.11 and removed in GitLab 14.0. Use the [DORA metrics API](dora/metrics.md) instead. diff --git a/doc/api/experiments.md b/doc/api/experiments.md index 669d00454bd..6393a358e51 100644 --- a/doc/api/experiments.md +++ b/doc/api/experiments.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Expansion +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/api/feature_flags.md b/doc/api/feature_flags.md index 0ad7b04d4f7..b00917674b0 100644 --- a/doc/api/feature_flags.md +++ b/doc/api/feature_flags.md @@ -144,7 +144,7 @@ POST /projects/:id/feature_flags | ------------------- | ---------------- | ---------- | ---------------------------------------------------------------------------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | | `name` | string | yes | The name of the feature flag. | -| `version` | string | yes | The version of the feature flag. Must be `new_version_flag`. Omit or set to `legacy_flag` to create a [Legacy Feature Flag](feature_flags_legacy.md). | +| `version` | string | yes | The version of the feature flag. Must be `new_version_flag`. Omit or set to `legacy_flag` to create a Legacy Feature Flag. | | `description` | string | no | The description of the feature flag. | | `active` | boolean | no | The active state of the flag. Defaults to true. [Supported](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/38350) in GitLab 13.3 and later. | | `strategies` | JSON | no | The feature flag [strategies](../operations/feature_flags.md#feature-flag-strategies). | diff --git a/doc/api/feature_flags_legacy.md b/doc/api/feature_flags_legacy.md deleted file mode 100644 index 1cf05144a1b..00000000000 --- a/doc/api/feature_flags_legacy.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'feature_flags.md' -remove_date: '2022-06-22' ---- - -This document was moved to [another location](feature_flags). - -<!-- This redirect file can be deleted after <2022-06-22>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/api/graphql/index.md b/doc/api/graphql/index.md index 09b97a78e04..be1bfc79aeb 100644 --- a/doc/api/graphql/index.md +++ b/doc/api/graphql/index.md @@ -61,14 +61,6 @@ You can work with sample queries that pull data from public projects on GitLab.c The [get started](getting_started.md) page includes different methods to customize GraphQL queries. -### Update the GraphQL API reference - -If you change the GraphQL schema, create a merge request to get your changes approved. -To generate the required documentation and schema, see -[Rake tasks for developers](../../development/rake_tasks.md#update-graphql-documentation-and-schema-definitions). - -Run the commands using the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit/). - ## Breaking changes The GitLab GraphQL API is [versionless](https://graphql.org/learn/best-practices/#versioning) and changes to the API are primarily backward-compatible. diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index 805f6a506b7..fbf6bc116f4 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -87,6 +87,16 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="queryciminutesusagenamespaceid"></a>`namespaceId` | [`NamespaceID`](#namespaceid) | Global ID of the Namespace for the monthly CI/CD minutes usage. | +### `Query.ciVariables` + +List of the instance's CI/CD variables. + +Returns [`CiVariableConnection`](#civariableconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + ### `Query.containerRepository` Find a container repository. @@ -377,6 +387,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="queryrunnersstatus"></a>`status` | [`CiRunnerStatus`](#cirunnerstatus) | Filter runners by status. | | <a id="queryrunnerstaglist"></a>`tagList` | [`[String!]`](#string) | Filter by tags associated with the runner (comma-separated or array). | | <a id="queryrunnerstype"></a>`type` | [`CiRunnerType`](#cirunnertype) | Filter runners by type. | +| <a id="queryrunnersupgradestatus"></a>`upgradeStatus` | [`CiRunnerUpgradeStatusType`](#cirunnerupgradestatustype) | Filter by upgrade status. | ### `Query.snippets` @@ -431,6 +442,18 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="querytimelogsstarttime"></a>`startTime` | [`Time`](#time) | List timelogs within a time range where the logged time is equal to or after startTime. | | <a id="querytimelogsusername"></a>`username` | [`String`](#string) | List timelogs for a user. | +### `Query.todo` + +Retrieve a single to-do item. + +Returns [`Todo`](#todo). + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="querytodoid"></a>`id` | [`TodoID!`](#todoid) | ID of the to-do item. | + ### `Query.topics` Find project topics. @@ -756,6 +779,27 @@ Input type: `AuditEventsStreamingHeadersDestroyInput` | <a id="mutationauditeventsstreamingheadersdestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationauditeventsstreamingheadersdestroyerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +### `Mutation.auditEventsStreamingHeadersUpdate` + +Input type: `AuditEventsStreamingHeadersUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationauditeventsstreamingheadersupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationauditeventsstreamingheadersupdateheaderid"></a>`headerId` | [`AuditEventsStreamingHeaderID!`](#auditeventsstreamingheaderid) | Header to update. | +| <a id="mutationauditeventsstreamingheadersupdatekey"></a>`key` | [`String!`](#string) | Header key. | +| <a id="mutationauditeventsstreamingheadersupdatevalue"></a>`value` | [`String!`](#string) | Header value. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationauditeventsstreamingheadersupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationauditeventsstreamingheadersupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationauditeventsstreamingheadersupdateheader"></a>`header` | [`AuditEventStreamingHeader`](#auditeventstreamingheader) | Updates header. | + ### `Mutation.awardEmojiAdd` Input type: `AwardEmojiAddInput` @@ -1391,6 +1435,7 @@ Input type: `CreateEpicInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationcreateepicaddlabelids"></a>`addLabelIds` | [`[ID!]`](#id) | IDs of labels to be added to the epic. | +| <a id="mutationcreateepicaddlabels"></a>`addLabels` | [`[String!]`](#string) | Array of labels to be added to the epic. | | <a id="mutationcreateepicclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationcreateepiccolor"></a>`color` | [`Color`](#color) | Color of the epic. Available only when feature flag `epic_color_highlight` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. | | <a id="mutationcreateepicconfidential"></a>`confidential` | [`Boolean`](#boolean) | Indicates if the epic is confidential. | @@ -2904,6 +2949,25 @@ Input type: `IssuableResourceLinkCreateInput` | <a id="mutationissuableresourcelinkcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationissuableresourcelinkcreateissuableresourcelink"></a>`issuableResourceLink` | [`IssuableResourceLink`](#issuableresourcelink) | Issuable resource link. | +### `Mutation.issuableResourceLinkDestroy` + +Input type: `IssuableResourceLinkDestroyInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationissuableresourcelinkdestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationissuableresourcelinkdestroyid"></a>`id` | [`IncidentManagementIssuableResourceLinkID!`](#incidentmanagementissuableresourcelinkid) | Issuable resource link ID to remove. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationissuableresourcelinkdestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationissuableresourcelinkdestroyerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationissuableresourcelinkdestroyissuableresourcelink"></a>`issuableResourceLink` | [`IssuableResourceLink`](#issuableresourcelink) | Issuable resource link. | + ### `Mutation.issueMove` Input type: `IssueMoveInput` @@ -3782,6 +3846,25 @@ Input type: `MergeRequestUpdateInput` | <a id="mutationmergerequestupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationmergerequestupdatemergerequest"></a>`mergeRequest` | [`MergeRequest`](#mergerequest) | Merge request after mutation. | +### `Mutation.namespaceBanDestroy` + +Input type: `NamespaceBanDestroyInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationnamespacebandestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationnamespacebandestroyid"></a>`id` | [`NamespacesNamespaceBanID!`](#namespacesnamespacebanid) | Global ID of the namespace ban to remove. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationnamespacebandestroyclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationnamespacebandestroyerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationnamespacebandestroynamespaceban"></a>`namespaceBan` | [`NamespaceBan`](#namespaceban) | Namespace Ban. | + ### `Mutation.namespaceCiCdSettingsUpdate` Input type: `NamespaceCiCdSettingsUpdateInput` @@ -3958,6 +4041,25 @@ Input type: `OncallScheduleUpdateInput` | <a id="mutationoncallscheduleupdateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationoncallscheduleupdateoncallschedule"></a>`oncallSchedule` | [`IncidentManagementOncallSchedule`](#incidentmanagementoncallschedule) | On-call schedule. | +### `Mutation.pagesMarkOnboardingComplete` + +Input type: `PagesMarkOnboardingCompleteInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationpagesmarkonboardingcompleteclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationpagesmarkonboardingcompleteprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationpagesmarkonboardingcompleteclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationpagesmarkonboardingcompleteerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationpagesmarkonboardingcompleteonboardingcomplete"></a>`onboardingComplete` | [`Boolean!`](#boolean) | Indicates the new onboarding_complete state of the project's Pages metadata. | + ### `Mutation.pipelineCancel` Input type: `PipelineCancelInput` @@ -5016,6 +5118,7 @@ Input type: `UpdateEpicInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationupdateepicaddlabelids"></a>`addLabelIds` | [`[ID!]`](#id) | IDs of labels to be added to the epic. | +| <a id="mutationupdateepicaddlabels"></a>`addLabels` | [`[String!]`](#string) | Array of labels to be added to the epic. | | <a id="mutationupdateepicclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationupdateepiccolor"></a>`color` | [`Color`](#color) | Color of the epic. Available only when feature flag `epic_color_highlight` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. | | <a id="mutationupdateepicconfidential"></a>`confidential` | [`Boolean`](#boolean) | Indicates if the epic is confidential. | @@ -5025,6 +5128,7 @@ Input type: `UpdateEpicInput` | <a id="mutationupdateepicgrouppath"></a>`groupPath` | [`ID!`](#id) | Group the epic to mutate is in. | | <a id="mutationupdateepiciid"></a>`iid` | [`ID!`](#id) | IID of the epic to mutate. | | <a id="mutationupdateepicremovelabelids"></a>`removeLabelIds` | [`[ID!]`](#id) | IDs of labels to be removed from the epic. | +| <a id="mutationupdateepicremovelabels"></a>`removeLabels` | [`[String!]`](#string) | Array of labels to be removed from the epic. | | <a id="mutationupdateepicstartdatefixed"></a>`startDateFixed` | [`String`](#string) | Start date of the epic. | | <a id="mutationupdateepicstartdateisfixed"></a>`startDateIsFixed` | [`Boolean`](#boolean) | Indicates start date should be sourced from start_date_fixed field not the issue milestones. | | <a id="mutationupdateepicstateevent"></a>`stateEvent` | [`EpicStateEvent`](#epicstateevent) | State event for the epic. | @@ -5419,7 +5523,8 @@ Input type: `VulnerabilityFindingDismissInput` | <a id="mutationvulnerabilityfindingdismissclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationvulnerabilityfindingdismisscomment"></a>`comment` | [`String`](#string) | Comment why finding should be dismissed. | | <a id="mutationvulnerabilityfindingdismissdismissalreason"></a>`dismissalReason` | [`VulnerabilityDismissalReason`](#vulnerabilitydismissalreason) | Reason why finding should be dismissed. | -| <a id="mutationvulnerabilityfindingdismissid"></a>`id` | [`VulnerabilitiesFindingID!`](#vulnerabilitiesfindingid) | ID of the finding to be dismissed. | +| <a id="mutationvulnerabilityfindingdismissid"></a>`id` **{warning-solid}** | [`VulnerabilitiesFindingID`](#vulnerabilitiesfindingid) | **Deprecated:** Use `uuid`. Deprecated in 15.2. | +| <a id="mutationvulnerabilityfindingdismissuuid"></a>`uuid` | [`String`](#string) | UUID of the finding to be dismissed. | #### Fields @@ -5483,6 +5588,7 @@ Input type: `WorkItemCreateInput` | ---- | ---- | ----------- | | <a id="mutationworkitemcreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationworkitemcreatedescription"></a>`description` | [`String`](#string) | Description of the work item. | +| <a id="mutationworkitemcreatehierarchywidget"></a>`hierarchyWidget` | [`WorkItemWidgetHierarchyCreateInput`](#workitemwidgethierarchycreateinput) | Input for hierarchy widget. | | <a id="mutationworkitemcreateprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project the work item is associated with. | | <a id="mutationworkitemcreatetitle"></a>`title` | [`String!`](#string) | Title of the work item. | | <a id="mutationworkitemcreateworkitemtypeid"></a>`workItemTypeId` | [`WorkItemsTypeID!`](#workitemstypeid) | Global ID of a work item type. | @@ -5589,9 +5695,12 @@ Input type: `WorkItemUpdateInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationworkitemupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationworkitemupdatedescriptionwidget"></a>`descriptionWidget` | [`WorkItemWidgetDescriptionInput`](#workitemwidgetdescriptioninput) | Input for description widget. | +| <a id="mutationworkitemupdatehierarchywidget"></a>`hierarchyWidget` | [`WorkItemWidgetHierarchyUpdateInput`](#workitemwidgethierarchyupdateinput) | Input for hierarchy widget. | | <a id="mutationworkitemupdateid"></a>`id` | [`WorkItemID!`](#workitemid) | Global ID of the work item. | | <a id="mutationworkitemupdatestateevent"></a>`stateEvent` | [`WorkItemStateEvent`](#workitemstateevent) | Close or reopen a work item. | | <a id="mutationworkitemupdatetitle"></a>`title` | [`String`](#string) | Title of the work item. | +| <a id="mutationworkitemupdateweightwidget"></a>`weightWidget` | [`WorkItemWidgetWeightInput`](#workitemwidgetweightinput) | Input for weight widget. | #### Fields @@ -6216,6 +6325,29 @@ The edge type for [`CiStage`](#cistage). | <a id="cistageedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="cistageedgenode"></a>`node` | [`CiStage`](#cistage) | The item at the end of the edge. | +#### `CiVariableConnection` + +The connection type for [`CiVariable`](#civariable). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="civariableconnectionedges"></a>`edges` | [`[CiVariableEdge]`](#civariableedge) | A list of edges. | +| <a id="civariableconnectionnodes"></a>`nodes` | [`[CiVariable]`](#civariable) | A list of nodes. | +| <a id="civariableconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `CiVariableEdge` + +The edge type for [`CiVariable`](#civariable). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="civariableedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="civariableedgenode"></a>`node` | [`CiVariable`](#civariable) | The item at the end of the edge. | + #### `ClusterAgentActivityEventConnection` The connection type for [`ClusterAgentActivityEvent`](#clusteragentactivityevent). @@ -7166,6 +7298,29 @@ The edge type for [`IncidentManagementOncallShift`](#incidentmanagementoncallshi | <a id="incidentmanagementoncallshiftedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="incidentmanagementoncallshiftedgenode"></a>`node` | [`IncidentManagementOncallShift`](#incidentmanagementoncallshift) | The item at the end of the edge. | +#### `IssuableResourceLinkConnection` + +The connection type for [`IssuableResourceLink`](#issuableresourcelink). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="issuableresourcelinkconnectionedges"></a>`edges` | [`[IssuableResourceLinkEdge]`](#issuableresourcelinkedge) | A list of edges. | +| <a id="issuableresourcelinkconnectionnodes"></a>`nodes` | [`[IssuableResourceLink]`](#issuableresourcelink) | A list of nodes. | +| <a id="issuableresourcelinkconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `IssuableResourceLinkEdge` + +The edge type for [`IssuableResourceLink`](#issuableresourcelink). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="issuableresourcelinkedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="issuableresourcelinkedgenode"></a>`node` | [`IssuableResourceLink`](#issuableresourcelink) | The item at the end of the edge. | + #### `IssueConnection` The connection type for [`Issue`](#issue). @@ -9422,7 +9577,7 @@ Represents an epic on an issue board. | <a id="boardepicblockedbyepics"></a>`blockedByEpics` | [`EpicConnection`](#epicconnection) | Epics blocking this epic. (see [Connections](#connections)) | | <a id="boardepicblockingcount"></a>`blockingCount` | [`Int`](#int) | Count of epics that this epic is blocking. | | <a id="boardepicclosedat"></a>`closedAt` | [`Time`](#time) | Timestamp of when the epic was closed. | -| <a id="boardepiccolor"></a>`color` | [`String!`](#string) | Color of the epic. Available only when feature flag `epic_color_highlight` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. | +| <a id="boardepiccolor"></a>`color` | [`String`](#string) | Color of the epic. Returns `null` if `epic_color_highlight` feature flag is disabled. | | <a id="boardepicconfidential"></a>`confidential` | [`Boolean`](#boolean) | Indicates if the epic is confidential. | | <a id="boardepiccreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of when the epic was created. | | <a id="boardepicdescendantcounts"></a>`descendantCounts` | [`EpicDescendantCount`](#epicdescendantcount) | Number of open and closed descendant epics and issues. | @@ -9458,7 +9613,7 @@ Represents an epic on an issue board. | <a id="boardepicstartdateisfixed"></a>`startDateIsFixed` | [`Boolean`](#boolean) | Indicates if the start date has been manually set. | | <a id="boardepicstate"></a>`state` | [`EpicState!`](#epicstate) | State of the epic. | | <a id="boardepicsubscribed"></a>`subscribed` | [`Boolean!`](#boolean) | Indicates the currently logged in user is subscribed to the epic. | -| <a id="boardepictextcolor"></a>`textColor` | [`String!`](#string) | Text color generated for the epic. Available only when feature flag `epic_color_highlight` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. | +| <a id="boardepictextcolor"></a>`textColor` | [`String`](#string) | Text color generated for the epic. Returns `null` if `epic_color_highlight` feature flag is disabled. | | <a id="boardepictitle"></a>`title` | [`String`](#string) | Title of the epic. | | <a id="boardepictitlehtml"></a>`titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title`. | | <a id="boardepicupdatedat"></a>`updatedAt` | [`Time`](#time) | Timestamp of when the epic was updated. | @@ -9782,6 +9937,7 @@ Represents the total number of issues and their weights for a particular day. | <a id="cijobid"></a>`id` | [`JobID`](#jobid) | ID of the job. | | <a id="cijobkind"></a>`kind` | [`CiJobKind!`](#cijobkind) | Indicates the type of job. | | <a id="cijobmanualjob"></a>`manualJob` | [`Boolean`](#boolean) | Whether the job has a manual action. | +| <a id="cijobmanualvariables"></a>`manualVariables` | [`CiVariableConnection`](#civariableconnection) | Variables added to a manual job when the job is triggered. (see [Connections](#connections)) | | <a id="cijobname"></a>`name` | [`String`](#string) | Name of the job. | | <a id="cijobneeds"></a>`needs` | [`CiBuildNeedConnection`](#cibuildneedconnection) | References to builds that must complete before the jobs run. (see [Connections](#connections)) | | <a id="cijobpipeline"></a>`pipeline` | [`Pipeline`](#pipeline) | Pipeline the job belongs to. | @@ -9791,6 +9947,7 @@ Represents the total number of issues and their weights for a particular day. | <a id="cijobqueuedduration"></a>`queuedDuration` | [`Duration`](#duration) | How long the job was enqueued before starting. | | <a id="cijobrefname"></a>`refName` | [`String`](#string) | Ref name of the job. | | <a id="cijobrefpath"></a>`refPath` | [`String`](#string) | Path to the ref. | +| <a id="cijobretried"></a>`retried` | [`Boolean`](#boolean) | Indicates that the job has been retried. | | <a id="cijobretryable"></a>`retryable` | [`Boolean!`](#boolean) | Indicates the job can be retried. | | <a id="cijobscheduledat"></a>`scheduledAt` | [`Time`](#time) | Schedule for the build. | | <a id="cijobschedulingtype"></a>`schedulingType` | [`String`](#string) | Type of job scheduling. Value is `dag` if the job uses the `needs` keyword, and `stage` otherwise. | @@ -9937,6 +10094,21 @@ GitLab CI/CD configuration template. | <a id="citemplatecontent"></a>`content` | [`String!`](#string) | Contents of the CI template. | | <a id="citemplatename"></a>`name` | [`String!`](#string) | Name of the CI template. | +### `CiVariable` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="civariableenvironmentscope"></a>`environmentScope` | [`String`](#string) | Scope defining the environments in which the variable can be used. | +| <a id="civariableid"></a>`id` | [`ID!`](#id) | ID of the variable. | +| <a id="civariablekey"></a>`key` | [`String`](#string) | Name of the variable. | +| <a id="civariablemasked"></a>`masked` | [`Boolean`](#boolean) | Indicates whether the variable is masked. | +| <a id="civariableprotected"></a>`protected` | [`Boolean`](#boolean) | Indicates whether the variable is protected. | +| <a id="civariableraw"></a>`raw` | [`Boolean`](#boolean) | Indicates whether the variable is raw. | +| <a id="civariablevalue"></a>`value` | [`String`](#string) | Value of the variable. | +| <a id="civariablevariabletype"></a>`variableType` | [`CiVariableType`](#civariabletype) | Type of the variable. | + ### `ClusterAgent` #### Fields @@ -10950,7 +11122,8 @@ Returns [`[DoraMetric!]`](#dorametric). | Name | Type | Description | | ---- | ---- | ----------- | | <a id="dorametricsenddate"></a>`endDate` | [`Date`](#date) | Date range to end at. Default is the current date. | -| <a id="dorametricsenvironmenttier"></a>`environmentTier` | [`DeploymentTier`](#deploymenttier) | Deployment tier of the environments to return. Defaults to `PRODUCTION`. | +| <a id="dorametricsenvironmenttier"></a>`environmentTier` | [`DeploymentTier`](#deploymenttier) | Deployment tier of the environments to return. Deprecated, please update to `environment_tiers` param. | +| <a id="dorametricsenvironmenttiers"></a>`environmentTiers` | [`[DeploymentTier!]`](#deploymenttier) | Deployment tiers of the environments to return. Defaults to [`PRODUCTION`]. | | <a id="dorametricsinterval"></a>`interval` | [`DoraMetricBucketingInterval`](#dorametricbucketinginterval) | How the metric should be aggregrated. Defaults to `DAILY`. In the case of `ALL`, the `date` field in the response will be `null`. | | <a id="dorametricsmetric"></a>`metric` | [`DoraMetricType!`](#dorametrictype) | Type of metric to return. | | <a id="dorametricsstartdate"></a>`startDate` | [`Date`](#date) | Date range to start from. Default is 3 months ago. | @@ -11007,7 +11180,7 @@ Represents an epic. | <a id="epicblockedbyepics"></a>`blockedByEpics` | [`EpicConnection`](#epicconnection) | Epics blocking this epic. (see [Connections](#connections)) | | <a id="epicblockingcount"></a>`blockingCount` | [`Int`](#int) | Count of epics that this epic is blocking. | | <a id="epicclosedat"></a>`closedAt` | [`Time`](#time) | Timestamp of when the epic was closed. | -| <a id="epiccolor"></a>`color` | [`String!`](#string) | Color of the epic. Available only when feature flag `epic_color_highlight` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. | +| <a id="epiccolor"></a>`color` | [`String`](#string) | Color of the epic. Returns `null` if `epic_color_highlight` feature flag is disabled. | | <a id="epicconfidential"></a>`confidential` | [`Boolean`](#boolean) | Indicates if the epic is confidential. | | <a id="epiccreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of when the epic was created. | | <a id="epicdescendantcounts"></a>`descendantCounts` | [`EpicDescendantCount`](#epicdescendantcount) | Number of open and closed descendant epics and issues. | @@ -11043,7 +11216,7 @@ Represents an epic. | <a id="epicstartdateisfixed"></a>`startDateIsFixed` | [`Boolean`](#boolean) | Indicates if the start date has been manually set. | | <a id="epicstate"></a>`state` | [`EpicState!`](#epicstate) | State of the epic. | | <a id="epicsubscribed"></a>`subscribed` | [`Boolean!`](#boolean) | Indicates the currently logged in user is subscribed to the epic. | -| <a id="epictextcolor"></a>`textColor` | [`String!`](#string) | Text color generated for the epic. Available only when feature flag `epic_color_highlight` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. | +| <a id="epictextcolor"></a>`textColor` | [`String`](#string) | Text color generated for the epic. Returns `null` if `epic_color_highlight` feature flag is disabled. | | <a id="epictitle"></a>`title` | [`String`](#string) | Title of the epic. | | <a id="epictitlehtml"></a>`titleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `title`. | | <a id="epicupdatedat"></a>`updatedAt` | [`Time`](#time) | Timestamp of when the epic was updated. | @@ -11322,6 +11495,22 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="epicissuecurrentusertodosstate"></a>`state` | [`TodoStateEnum`](#todostateenum) | State of the to-do items. | +##### `EpicIssue.issuableResourceLinks` + +Issuable resource links of the incident issue. + +Returns [`IssuableResourceLinkConnection`](#issuableresourcelinkconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="epicissueissuableresourcelinksincidentid"></a>`incidentId` | [`IssueID!`](#issueid) | ID of the incident. | + ##### `EpicIssue.reference` Internal reference of the issue. Returned in shortened format by default. @@ -11448,7 +11637,7 @@ Represents an external resource to send audit events to. | ---- | ---- | ----------- | | <a id="externalauditeventdestinationdestinationurl"></a>`destinationUrl` | [`String!`](#string) | External destination to send audit events to. | | <a id="externalauditeventdestinationgroup"></a>`group` | [`Group!`](#group) | Group the destination belongs to. | -| <a id="externalauditeventdestinationheaders"></a>`headers` | [`AuditEventStreamingHeaderConnection!`](#auditeventstreamingheaderconnection) | List of additional HTTP headers sent with each event. Available only when feature flag `streaming_audit_event_headers` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. (see [Connections](#connections)) | +| <a id="externalauditeventdestinationheaders"></a>`headers` | [`AuditEventStreamingHeaderConnection!`](#auditeventstreamingheaderconnection) | List of additional HTTP headers sent with each event. Available only when feature flag `streaming_audit_event_headers` is enabled. This flag is enabled by default. (see [Connections](#connections)) | | <a id="externalauditeventdestinationid"></a>`id` | [`ID!`](#id) | ID of the destination. | | <a id="externalauditeventdestinationverificationtoken"></a>`verificationToken` | [`String!`](#string) | Verification token to validate source of event. | @@ -11675,6 +11864,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupallowstalerunnerpruning"></a>`allowStaleRunnerPruning` | [`Boolean!`](#boolean) | Indicates whether to regularly prune stale group runners. Defaults to false. | | <a id="groupautodevopsenabled"></a>`autoDevopsEnabled` | [`Boolean`](#boolean) | Indicates whether Auto DevOps is enabled for all projects within this group. | | <a id="groupavatarurl"></a>`avatarUrl` | [`String`](#string) | Avatar URL of the group. | +| <a id="groupcivariables"></a>`ciVariables` | [`CiVariableConnection`](#civariableconnection) | List of the group's CI/CD variables. (see [Connections](#connections)) | | <a id="groupcontainerrepositoriescount"></a>`containerRepositoriesCount` | [`Int!`](#int) | Number of container repositories in the group. | | <a id="groupcontainslockedprojects"></a>`containsLockedProjects` | [`Boolean!`](#boolean) | Includes at least one project where the repository size exceeds the limit. | | <a id="groupcrossprojectpipelineavailable"></a>`crossProjectPipelineAvailable` | [`Boolean!`](#boolean) | Indicates if the cross_project_pipeline feature is available for the namespace. | @@ -11691,6 +11881,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupdescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | <a id="groupdora"></a>`dora` | [`Dora`](#dora) | Group's DORA metrics. | | <a id="groupemailsdisabled"></a>`emailsDisabled` | [`Boolean`](#boolean) | Indicates if a group has email notifications disabled. | +| <a id="groupenforcefreeusercap"></a>`enforceFreeUserCap` | [`Boolean`](#boolean) | Indicates whether the group has limited users for a free plan. | | <a id="groupepicboards"></a>`epicBoards` | [`EpicBoardConnection`](#epicboardconnection) | Find epic boards. (see [Connections](#connections)) | | <a id="groupepicsenabled"></a>`epicsEnabled` | [`Boolean`](#boolean) | Indicates if Epics are enabled for namespace. | | <a id="groupexternalauditeventdestinations"></a>`externalAuditEventDestinations` | [`ExternalAuditEventDestinationConnection`](#externalauditeventdestinationconnection) | External locations that receive audit events belonging to the group. (see [Connections](#connections)) | @@ -11766,6 +11957,22 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="groupboardsid"></a>`id` | [`BoardID`](#boardid) | Find a board by its ID. | +##### `Group.clusterAgents` + +Cluster agents associated with projects in the group and its subgroups. + +Returns [`ClusterAgentConnection`](#clusteragentconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="groupclusteragentshasvulnerabilities"></a>`hasVulnerabilities` | [`Boolean`](#boolean) | Returns only cluster agents which have vulnerabilities. | + ##### `Group.codeCoverageActivities` Represents the code coverage activity for this group. @@ -12236,6 +12443,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="grouprunnersstatus"></a>`status` | [`CiRunnerStatus`](#cirunnerstatus) | Filter runners by status. | | <a id="grouprunnerstaglist"></a>`tagList` | [`[String!]`](#string) | Filter by tags associated with the runner (comma-separated or array). | | <a id="grouprunnerstype"></a>`type` | [`CiRunnerType`](#cirunnertype) | Filter runners by type. | +| <a id="grouprunnersupgradestatus"></a>`upgradeStatus` | [`CiRunnerUpgradeStatusType`](#cirunnerupgradestatustype) | Filter by upgrade status. | ##### `Group.scanExecutionPolicies` @@ -12559,6 +12767,22 @@ A block of time for which a participant is on-call. #### Fields with arguments +##### `InstanceSecurityDashboard.clusterAgents` + +Cluster agents associated with projects selected in the Instance Security Dashboard. + +Returns [`ClusterAgentConnection`](#clusteragentconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="instancesecuritydashboardclusteragentshasvulnerabilities"></a>`hasVulnerabilities` | [`Boolean`](#boolean) | Returns only cluster agents which have vulnerabilities. | + ##### `InstanceSecurityDashboard.projects` Projects selected in Instance Security Dashboard. @@ -12695,6 +12919,22 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="issuecurrentusertodosstate"></a>`state` | [`TodoStateEnum`](#todostateenum) | State of the to-do items. | +##### `Issue.issuableResourceLinks` + +Issuable resource links of the incident issue. + +Returns [`IssuableResourceLinkConnection`](#issuableresourcelinkconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="issueissuableresourcelinksincidentid"></a>`incidentId` | [`IssueID!`](#issueid) | ID of the incident. | + ##### `Issue.reference` Internal reference of the issue. Returned in shortened format by default. @@ -14289,6 +14529,16 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="namespacescanexecutionpoliciesactionscantypes"></a>`actionScanTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filters policies by the action scan type. Only these scan types are supported: `dast`, `secret_detection`, `cluster_image_scanning`, `container_scanning`, `sast`. | | <a id="namespacescanexecutionpoliciesrelationship"></a>`relationship` | [`SecurityPolicyRelationType`](#securitypolicyrelationtype) | Filter policies by the given policy relationship. | +### `NamespaceBan` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="namespacebanid"></a>`id` | [`GlobalID!`](#globalid) | Global ID of the namespace ban. | +| <a id="namespacebannamespace"></a>`namespace` | [`Namespace!`](#namespace) | Root namespace to which the ban applies. | +| <a id="namespacebanuser"></a>`user` | [`UserCore!`](#usercore) | User to which the namespace ban applies. | + ### `NamespaceCiCdSetting` #### Fields @@ -14757,6 +15007,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="pipelinejobsretried"></a>`retried` | [`Boolean`](#boolean) | Filter jobs by retry-status. | | <a id="pipelinejobssecurityreporttypes"></a>`securityReportTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filter jobs by the type of security report they produce. | | <a id="pipelinejobsstatuses"></a>`statuses` | [`[CiJobStatus!]`](#cijobstatus) | Filter jobs by status. | @@ -14913,7 +15164,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectcicdsettings"></a>`ciCdSettings` | [`ProjectCiCdSetting`](#projectcicdsetting) | CI/CD settings for the project. | | <a id="projectciconfigpathordefault"></a>`ciConfigPathOrDefault` | [`String!`](#string) | Path of the CI configuration file. | | <a id="projectcijobtokenscope"></a>`ciJobTokenScope` | [`CiJobTokenScopeType`](#cijobtokenscopetype) | The CI Job Tokens scope of access. | -| <a id="projectclusteragents"></a>`clusterAgents` | [`ClusterAgentConnection`](#clusteragentconnection) | Cluster agents associated with the project. (see [Connections](#connections)) | +| <a id="projectcivariables"></a>`ciVariables` | [`CiVariableConnection`](#civariableconnection) | List of the project's CI/CD variables. (see [Connections](#connections)) | | <a id="projectcodecoveragesummary"></a>`codeCoverageSummary` | [`CodeCoverageSummary`](#codecoveragesummary) | Code coverage summary associated with the project. | | <a id="projectcomplianceframeworks"></a>`complianceFrameworks` | [`ComplianceFrameworkConnection`](#complianceframeworkconnection) | Compliance frameworks associated with the project. (see [Connections](#connections)) | | <a id="projectcontainerexpirationpolicy"></a>`containerExpirationPolicy` | [`ContainerExpirationPolicy`](#containerexpirationpolicy) | Container expiration policy of the project. | @@ -14966,8 +15217,8 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectsecuritydashboardpath"></a>`securityDashboardPath` | [`String`](#string) | Path to project's security dashboard. | | <a id="projectsecurityscanners"></a>`securityScanners` | [`SecurityScanners`](#securityscanners) | Information about security analyzers used in the project. | | <a id="projectsentryerrors"></a>`sentryErrors` | [`SentryErrorCollection`](#sentryerrorcollection) | Paginated collection of Sentry errors on the project. | -| <a id="projectservicedeskaddress"></a>`serviceDeskAddress` | [`String`](#string) | E-mail address of the service desk. | -| <a id="projectservicedeskenabled"></a>`serviceDeskEnabled` | [`Boolean`](#boolean) | Indicates if the project has service desk enabled. | +| <a id="projectservicedeskaddress"></a>`serviceDeskAddress` | [`String`](#string) | E-mail address of the Service Desk. | +| <a id="projectservicedeskenabled"></a>`serviceDeskEnabled` | [`Boolean`](#boolean) | Indicates if the project has Service Desk enabled. | | <a id="projectsharedrunnersenabled"></a>`sharedRunnersEnabled` | [`Boolean`](#boolean) | Indicates if shared runners are enabled for the project. | | <a id="projectsnippetsenabled"></a>`snippetsEnabled` | [`Boolean`](#boolean) | Indicates if Snippets are enabled for the current user. | | <a id="projectsquashcommittemplate"></a>`squashCommitTemplate` | [`String`](#string) | Template used to create squash commit message in merge requests. | @@ -15133,8 +15384,25 @@ Returns [`ClusterAgent`](#clusteragent). | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="projectclusteragenthasvulnerabilities"></a>`hasVulnerabilities` | [`Boolean`](#boolean) | Returns only cluster agents which have vulnerabilities. | | <a id="projectclusteragentname"></a>`name` | [`String!`](#string) | Name of the cluster agent. | +##### `Project.clusterAgents` + +Cluster agents associated with the project. + +Returns [`ClusterAgentConnection`](#clusteragentconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectclusteragentshasvulnerabilities"></a>`hasVulnerabilities` | [`Boolean`](#boolean) | Returns only cluster agents which have vulnerabilities. | + ##### `Project.containerRepositories` Container repositories of the project. @@ -16206,6 +16474,7 @@ Represents a release. | <a id="releasedescription"></a>`description` | [`String`](#string) | Description (also known as "release notes") of the release. | | <a id="releasedescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | | <a id="releaseevidences"></a>`evidences` | [`ReleaseEvidenceConnection`](#releaseevidenceconnection) | Evidence for the release. (see [Connections](#connections)) | +| <a id="releasehistoricalrelease"></a>`historicalRelease` | [`Boolean`](#boolean) | Indicates the release is an historical release. | | <a id="releaseid"></a>`id` | [`ReleaseID!`](#releaseid) | Global ID of the release. | | <a id="releaselinks"></a>`links` | [`ReleaseLinks`](#releaselinks) | Links of the release. | | <a id="releasemilestones"></a>`milestones` | [`MilestoneConnection`](#milestoneconnection) | Milestones associated to the release. (see [Connections](#connections)) | @@ -16628,8 +16897,10 @@ Represents the scan result policy. | ---- | ---- | ----------- | | <a id="scanresultpolicydescription"></a>`description` | [`String!`](#string) | Description of the policy. | | <a id="scanresultpolicyenabled"></a>`enabled` | [`Boolean!`](#boolean) | Indicates whether this policy is enabled. | +| <a id="scanresultpolicygroupapprovers"></a>`groupApprovers` | [`[Group!]`](#group) | Approvers of the group type. | | <a id="scanresultpolicyname"></a>`name` | [`String!`](#string) | Name of the policy. | | <a id="scanresultpolicyupdatedat"></a>`updatedAt` | [`Time!`](#time) | Timestamp of when the policy YAML was last updated. | +| <a id="scanresultpolicyuserapprovers"></a>`userApprovers` | [`[UserCore!]`](#usercore) | Approvers of the user type. | | <a id="scanresultpolicyyaml"></a>`yaml` | [`String!`](#string) | YAML definition of the policy. | ### `ScannedResource` @@ -18273,6 +18544,19 @@ Check permissions for the current user on a work item. | <a id="workitemtypeid"></a>`id` | [`WorkItemsTypeID!`](#workitemstypeid) | Global ID of the work item type. | | <a id="workitemtypename"></a>`name` | [`String!`](#string) | Name of the work item type. | +### `WorkItemWidgetAssignees` + +Represents an assignees widget. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgetassigneesallowsmultipleassignees"></a>`allowsMultipleAssignees` | [`Boolean`](#boolean) | Indicates whether multiple assignees are allowed. | +| <a id="workitemwidgetassigneesassignees"></a>`assignees` | [`UserCoreConnection`](#usercoreconnection) | Assignees of the work item. (see [Connections](#connections)) | +| <a id="workitemwidgetassigneescaninvitemembers"></a>`canInviteMembers` | [`Boolean!`](#boolean) | Indicates whether the current user can invite members to the work item's project. | +| <a id="workitemwidgetassigneestype"></a>`type` | [`WorkItemWidgetType`](#workitemwidgettype) | Widget type. | + ### `WorkItemWidgetDescription` Represents a description widget. @@ -18297,6 +18581,17 @@ Represents a hierarchy widget. | <a id="workitemwidgethierarchyparent"></a>`parent` | [`WorkItem`](#workitem) | Parent work item. | | <a id="workitemwidgethierarchytype"></a>`type` | [`WorkItemWidgetType`](#workitemwidgettype) | Widget type. | +### `WorkItemWidgetWeight` + +Represents a weight widget. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgetweighttype"></a>`type` | [`WorkItemWidgetType`](#workitemwidgettype) | Widget type. | +| <a id="workitemwidgetweightweight"></a>`weight` | [`Int`](#int) | Weight of the work item. | + ## Enumeration types Also called _Enums_, enumeration types are a special kind of scalar that @@ -18568,6 +18863,13 @@ Values for sorting runners. | <a id="cirunnerupgradestatustyperecommended"></a>`RECOMMENDED` | Upgrade is available and recommended for the runner. | | <a id="cirunnerupgradestatustypeunknown"></a>`UNKNOWN` | Upgrade status is unknown. | +### `CiVariableType` + +| Value | Description | +| ----- | ----------- | +| <a id="civariabletypeenv_var"></a>`ENV_VAR` | Env var type. | +| <a id="civariabletypefile"></a>`FILE` | File type. | + ### `CodeQualityDegradationSeverity` | Value | Description | @@ -19005,6 +19307,7 @@ User permission on groups. | Value | Description | | ----- | ----------- | | <a id="grouppermissioncreate_projects"></a>`CREATE_PROJECTS` | Groups where the user can create projects. | +| <a id="grouppermissiontransfer_projects"></a>`TRANSFER_PROJECTS` | Groups where the user can transfer projects to. | ### `HealthStatus` @@ -19148,6 +19451,7 @@ Issue type. | <a id="issuetypeincident"></a>`INCIDENT` | Incident issue type. | | <a id="issuetypeissue"></a>`ISSUE` | Issue issue type. | | <a id="issuetyperequirement"></a>`REQUIREMENT` | Requirement issue type. | +| <a id="issuetypetask"></a>`TASK` **{warning-solid}** | **Introduced** in 15.2. This feature is in Alpha. It can be changed or removed at any time. | | <a id="issuetypetest_case"></a>`TEST_CASE` | Test Case issue type. | ### `IterationSearchableField` @@ -19669,8 +19973,9 @@ The status of the security scan. | Value | Description | | ----- | ----------- | -| <a id="securitypolicyrelationtypedirect"></a>`DIRECT` | Policies defined for the project only. | -| <a id="securitypolicyrelationtypeinherited"></a>`INHERITED` | Policies defined for the project and project's ancestor groups. | +| <a id="securitypolicyrelationtypedirect"></a>`DIRECT` | Policies defined for the project/group only. | +| <a id="securitypolicyrelationtypeinherited"></a>`INHERITED` | Policies defined for the project/group and ancestor groups. | +| <a id="securitypolicyrelationtypeinherited_only"></a>`INHERITED_ONLY` | Policies defined for the project/group's ancestor groups only. | ### `SecurityReportTypeEnum` @@ -19911,7 +20216,7 @@ Name of the feature that the callout is for. | <a id="usercalloutfeaturenameenumgeo_migrate_hashed_storage"></a>`GEO_MIGRATE_HASHED_STORAGE` | Callout feature name for geo_migrate_hashed_storage. | | <a id="usercalloutfeaturenameenumgke_cluster_integration"></a>`GKE_CLUSTER_INTEGRATION` | Callout feature name for gke_cluster_integration. | | <a id="usercalloutfeaturenameenumgold_trial_billings"></a>`GOLD_TRIAL_BILLINGS` | Callout feature name for gold_trial_billings. | -| <a id="usercalloutfeaturenameenumminute_limit_banner"></a>`MINUTE_LIMIT_BANNER` | Callout feature name for minute_limit_banner. | +| <a id="usercalloutfeaturenameenummr_experience_survey"></a>`MR_EXPERIENCE_SURVEY` | Callout feature name for mr_experience_survey. | | <a id="usercalloutfeaturenameenumnew_user_signups_cap_reached"></a>`NEW_USER_SIGNUPS_CAP_REACHED` | Callout feature name for new_user_signups_cap_reached. | | <a id="usercalloutfeaturenameenumpersonal_access_token_expiry"></a>`PERSONAL_ACCESS_TOKEN_EXPIRY` | Callout feature name for personal_access_token_expiry. | | <a id="usercalloutfeaturenameenumpersonal_project_limitations_banner"></a>`PERSONAL_PROJECT_LIMITATIONS_BANNER` | Callout feature name for personal_project_limitations_banner. | @@ -20138,8 +20443,10 @@ Type of a work item widget. | Value | Description | | ----- | ----------- | +| <a id="workitemwidgettypeassignees"></a>`ASSIGNEES` | Assignees widget. | | <a id="workitemwidgettypedescription"></a>`DESCRIPTION` | Description widget. | | <a id="workitemwidgettypehierarchy"></a>`HIERARCHY` | Hierarchy widget. | +| <a id="workitemwidgettypeweight"></a>`WEIGHT` | Weight widget. | ## Scalar types @@ -20541,6 +20848,12 @@ A `NamespaceID` is a global ID. It is encoded as a string. An example `NamespaceID` is: `"gid://gitlab/Namespace/1"`. +### `NamespacesNamespaceBanID` + +A `NamespacesNamespaceBanID` is a global ID. It is encoded as a string. + +An example `NamespacesNamespaceBanID` is: `"gid://gitlab/Namespaces::NamespaceBan/1"`. + ### `NoteID` A `NoteID` is a global ID. It is encoded as a string. @@ -21358,8 +21671,10 @@ four standard [pagination arguments](#connection-pagination-arguments): Implementations: +- [`WorkItemWidgetAssignees`](#workitemwidgetassignees) - [`WorkItemWidgetDescription`](#workitemwidgetdescription) - [`WorkItemWidgetHierarchy`](#workitemwidgethierarchy) +- [`WorkItemWidgetWeight`](#workitemwidgetweight) ##### Fields @@ -21845,9 +22160,12 @@ A time-frame defined as a closed inclusive range of two dates. | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="workitemupdatedtaskinputdescriptionwidget"></a>`descriptionWidget` | [`WorkItemWidgetDescriptionInput`](#workitemwidgetdescriptioninput) | Input for description widget. | +| <a id="workitemupdatedtaskinputhierarchywidget"></a>`hierarchyWidget` | [`WorkItemWidgetHierarchyUpdateInput`](#workitemwidgethierarchyupdateinput) | Input for hierarchy widget. | | <a id="workitemupdatedtaskinputid"></a>`id` | [`WorkItemID!`](#workitemid) | Global ID of the work item. | | <a id="workitemupdatedtaskinputstateevent"></a>`stateEvent` | [`WorkItemStateEvent`](#workitemstateevent) | Close or reopen a work item. | | <a id="workitemupdatedtaskinputtitle"></a>`title` | [`String`](#string) | Title of the work item. | +| <a id="workitemupdatedtaskinputweightwidget"></a>`weightWidget` | [`WorkItemWidgetWeightInput`](#workitemwidgetweightinput) | Input for weight widget. | ### `WorkItemWidgetDescriptionInput` @@ -21856,3 +22174,28 @@ A time-frame defined as a closed inclusive range of two dates. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="workitemwidgetdescriptioninputdescription"></a>`description` | [`String!`](#string) | Description of the work item. | + +### `WorkItemWidgetHierarchyCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgethierarchycreateinputparentid"></a>`parentId` | [`WorkItemID`](#workitemid) | Global ID of the parent work item. | + +### `WorkItemWidgetHierarchyUpdateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgethierarchyupdateinputchildrenids"></a>`childrenIds` | [`[WorkItemID!]`](#workitemid) | Global IDs of children work items. | +| <a id="workitemwidgethierarchyupdateinputparentid"></a>`parentId` | [`WorkItemID`](#workitemid) | Global ID of the parent work item. Use `null` to remove the association. | + +### `WorkItemWidgetWeightInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="workitemwidgetweightinputweight"></a>`weight` | [`Int!`](#int) | Weight of the work item. | diff --git a/doc/api/groups.md b/doc/api/groups.md index 8372d6deddd..c51f23decb9 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -823,7 +823,7 @@ Parameters: | `subgroup_creation_level` | string | no | Allowed to [create subgroups](../user/group/subgroups/index.md#create-a-subgroup). Can be `owner` (Owners), or `maintainer` (users with the Maintainer role). | | `two_factor_grace_period` | integer | no | Time before Two-factor authentication is enforced (in hours). | | `visibility` | string | no | The group's visibility. Can be `private`, `internal`, or `public`. | -| `membership_lock` **(PREMIUM)** | boolean | no | Prevent adding new members to projects within this group. | +| `membership_lock` **(PREMIUM)** | boolean | no | Users cannot be added to projects in this group. | | `extra_shared_runners_minutes_limit` **(PREMIUM)** | integer | no | Can be set by administrators only. Additional CI/CD minutes for this group. | | `shared_runners_minutes_limit` **(PREMIUM)** | integer | no | Can be set by administrators only. Maximum number of monthly CI/CD minutes for this group. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | @@ -930,7 +930,7 @@ PUT /groups/:id | `visibility` | string | no | The visibility level of the group. Can be `private`, `internal`, or `public`. | | `extra_shared_runners_minutes_limit` **(PREMIUM)** | integer | no | Can be set by administrators only. Additional CI/CD minutes for this group. | | `file_template_project_id` **(PREMIUM)** | integer | no | The ID of a project to load custom file templates from. | -| `membership_lock` **(PREMIUM)** | boolean | no | Prevent adding new members to projects within this group. | +| `membership_lock` **(PREMIUM)** | boolean | no | Users cannot be added to projects in this group. | | `prevent_forking_outside_group` **(PREMIUM)** | boolean | no | When enabled, users can **not** fork projects from this group to external namespaces. | | `shared_runners_minutes_limit` **(PREMIUM)** | integer | no | Can be set by administrators only. Maximum number of monthly CI/CD minutes for this group. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | diff --git a/doc/api/index.md b/doc/api/index.md index c29e43d3f8e..26447a2223d 100644 --- a/doc/api/index.md +++ b/doc/api/index.md @@ -441,7 +441,7 @@ GitLab also returns the following additional pagination headers: | `x-next-page` | The index of the next page. | | `x-page` | The index of the current page (starting at 1). | | `x-per-page` | The number of items per page. | -| `X-prev-page` | The index of the previous page. | +| `x-prev-page` | The index of the previous page. | | `x-total` | The total number of items. | | `x-total-pages` | The total number of pages. | @@ -453,12 +453,14 @@ Keyset-pagination allows for more efficient retrieval of pages and - in contrast to offset-based pagination - runtime is independent of the size of the collection. -This method is controlled by the following parameters: +This method is controlled by the following parameters. `order_by` and `sort` are both mandatory. -| Parameter | Description | -|--------------| ------------| -| `pagination` | `keyset` (to enable keyset pagination). | -| `per_page` | Number of items to list per page (default: `20`, max: `100`). | +| Parameter | Required | Description | +|--------------| ------------ | --------- | +| `pagination` | yes | `keyset` (to enable keyset pagination). | +| `per_page` | no | Number of items to list per page (default: `20`, max: `100`). | +| `order_by` | yes | Column by which to order by. | +| `sort` | yes | Sort order (`asc` or `desc`) | In the following example, we list 50 [projects](projects.md) per page, ordered by `id` ascending. @@ -520,10 +522,20 @@ pagination headers. Keyset-based pagination is supported only for selected resources and ordering options: -| Resource | Options | Availability | -|:-------------------------|:---------------------------------|:----------------------------------------| -| [Projects](projects.md) | `order_by=id` only | Authenticated and unauthenticated users | -| [Groups](groups.md) | `order_by=name`, `sort=asc` only | Unauthenticated users only | +| Resource | Options | Availability | +|:---------------------------------------------------------|:---------------------------------|:------------------------------------------------------------------------------------------------------------| +| [Projects](projects.md) | `order_by=id` only | Authenticated and unauthenticated users | +| [Groups](groups.md) | `order_by=name`, `sort=asc` only | Unauthenticated users only | +| [Group audit events](audit_events.md#group-audit-events) | `order_by=id`, `sort=desc` only | Authenticated users only ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333968) in GitLab 15.2 | + +### Pagination response headers + +For performance reasons, if a query returns more than 10,000 records, GitLab +doesn't return the following headers: + +- `x-total`. +- `x-total-pages`. +- `rel="last"` `link` ## Path parameters diff --git a/doc/api/integrations.md b/doc/api/integrations.md index eaa826b3686..fca1d02161b 100644 --- a/doc/api/integrations.md +++ b/doc/api/integrations.md @@ -1033,7 +1033,6 @@ Parameters: | Parameter | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `recipients` | string | yes | Comma-separated list of recipient email addresses | -| `add_pusher` | boolean | no | Add pusher to recipients list | | `notify_only_broken_pipelines` | boolean | no | Notify only broken pipelines | | `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected. The default value is "default" | | `notify_only_default_branch` | boolean | no | Send notifications only for the default branch ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28271)) | diff --git a/doc/api/invitations.md b/doc/api/invitations.md index 5a39a86d039..96c820536de 100644 --- a/doc/api/invitations.md +++ b/doc/api/invitations.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Expansion +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -20,7 +20,7 @@ levels are defined in the `Gitlab::Access` module. Currently, these levels are v - Reporter (`20`) - Developer (`30`) - Maintainer (`40`) -- Owner (`50`) - Only valid to set for groups +- Owner (`50`). Valid for projects in [GitLab 14.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/21432). NOTE: From [GitLab 14.9](https://gitlab.com/gitlab-org/gitlab/-/issues/351211) and later, projects have a maximum role of Owner. diff --git a/doc/api/jobs.md b/doc/api/jobs.md index 85cdf7d892a..b23c33ddc0d 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -692,7 +692,7 @@ Example of response "finished_at": null, "duration": 8, "queued_duration": 0.010, - "id": 42, + "id": 1, "name": "rubocop", "ref": "main", "artifacts": [], @@ -742,7 +742,7 @@ Example of response "finished_at": null, "duration": null, "queued_duration": 0.010, - "id": 42, + "id": 1, "name": "rubocop", "ref": "main", "artifacts": [], @@ -792,7 +792,7 @@ Example of response "coverage": null, "allow_failure": false, "download_url": null, - "id": 42, + "id": 1, "name": "rubocop", "ref": "main", "artifacts": [], @@ -873,7 +873,7 @@ Example response: "finished_at": null, "duration": null, "queued_duration": 0.010, - "id": 42, + "id": 1, "name": "rubocop", "ref": "main", "artifacts": [], diff --git a/doc/api/members.md b/doc/api/members.md index 5002e1003e3..a9817918d0b 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -18,7 +18,7 @@ The access levels are defined in the `Gitlab::Access` module. Currently, these l - Reporter (`20`) - Developer (`30`) - Maintainer (`40`) -- Owner (`50`) - Only valid to set for groups +- Owner (`50`). Valid for projects in [GitLab 14.9 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/21432). NOTE: In [GitLab 14.9](https://gitlab.com/gitlab-org/gitlab/-/issues/351211) and later, projects in personal namespaces have an `access_level` of `50`(Owner). diff --git a/doc/api/merge_requests.md b/doc/api/merge_requests.md index eb4b7a3dd7e..c6714459643 100644 --- a/doc/api/merge_requests.md +++ b/doc/api/merge_requests.md @@ -450,6 +450,7 @@ Parameters: | `assignee_id` | integer | no | Returns merge requests assigned to the given user `id`. `None` returns unassigned merge requests. `Any` returns merge requests with an assignee. | | `approver_ids` **(PREMIUM)** | integer array | no | Returns merge requests which have specified all the users with the given `id`s as individual approvers. `None` returns merge requests without approvers. `Any` returns merge requests with an approver. | | `approved_by_ids` **(PREMIUM)** | integer array | no | Returns merge requests which have been approved by all the users with the given `id`s (Max: 5). `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | +| `approved_by_usernames` **(PREMIUM)** | string array | no | Returns merge requests which have been approved by all the users with the given `username`s (Max: 5). `None` returns merge requests with no approvals. `Any` returns merge requests with an approval. | | `reviewer_id` | integer | no | Returns merge requests which have the user as a [reviewer](../user/project/merge_requests/getting_started.md#reviewer) with the given user `id`. `None` returns merge requests with no reviewers. `Any` returns merge requests with any reviewer. Mutually exclusive with `reviewer_username`. | | `reviewer_username` | string | no | Returns merge requests which have the user as a [reviewer](../user/project/merge_requests/getting_started.md#reviewer) with the given `username`. `None` returns merge requests with no reviewers. `Any` returns merge requests with any reviewer. Mutually exclusive with `reviewer_id`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49341) in GitLab 13.8. | | `my_reaction_emoji` | string | no | Return merge requests reacted by the authenticated user by the given `emoji`. `None` returns issues not given a reaction. `Any` returns issues given at least one reaction. | @@ -785,6 +786,8 @@ the `approvals_before_merge` parameter: field `merge_user` can be either user who merged this merge request, user who set it to merge when pipeline succeeds or `null`. Field `merged_by` (user who merged this merge request or `null`) has been deprecated. +- `pipeline` is an old parameter and should not be used. Use `head_pipeline` instead, + as it is faster and returns more information. ## Get single MR participants diff --git a/doc/api/metadata.md b/doc/api/metadata.md new file mode 100644 index 00000000000..70c29ef5748 --- /dev/null +++ b/doc/api/metadata.md @@ -0,0 +1,46 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Metadata API **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357032) in GitLab 15.2. + +Retrieve metadata information for this GitLab instance. + +```plaintext +GET /metadata +``` + +Response body attributes: + +| Attribute | Type | Description | +|:------------------|:---------------|:-----------------------------------------------------------------------------------------| +| `version` | string | Version of the GitLab instance. | +| `revision` | string | Revision of the GitLab instance. | +| `kas` | object | Metadata about the GitLab agent server for Kubernetes (KAS). | +| `kas.enabled` | boolean | Indicates whether KAS is enabled. | +| `kas.externalUrl` | string or null | URL used by the agents to communicate with KAS. It's `null` if `kas.enabled` is `false`. | +| `kas.version` | string or null | Version of KAS. It's `null` if `kas.enabled` is `false`. | + +Example request: + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/metadata" +``` + +Example response: + +```json +{ + "version": "15.2-pre", + "revision": "c401a659d0c", + "kas": { + "enabled": true, + "externalUrl": "grpc://gitlab.example.com:8150", + "version": "15.0.0" + } +} +``` diff --git a/doc/api/metrics_user_starred_dashboards.md b/doc/api/metrics_user_starred_dashboards.md index 3e54ec74b24..4f5fe1c909a 100644 --- a/doc/api/metrics_user_starred_dashboards.md +++ b/doc/api/metrics_user_starred_dashboards.md @@ -12,7 +12,7 @@ by displaying favorited dashboards at the top of the select list. ## Add a star to a dashboard -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31316) in GitLab 13.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/31316) in GitLab 13.0. ```plaintext POST /projects/:id/metrics/user_starred_dashboards diff --git a/doc/api/notes.md b/doc/api/notes.md index fbcf5e28f79..f7caae59b4d 100644 --- a/doc/api/notes.md +++ b/doc/api/notes.md @@ -146,7 +146,7 @@ Parameters: | `issue_iid` | integer | yes | The IID of an issue. | | `body` | string | yes | The content of a note. Limited to 1,000,000 characters. | | `confidential` | boolean | no | The confidential flag of a note. Default is false. | -| `created_at` | string | no | Date time string, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | +| `created_at` | string | no | Date time string, ISO 8601 formatted. It must be after 1970-01-01. Example: `2016-03-11T03:45:40Z` (requires administrator or project/group owner rights) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/issues/11/notes?body=note" diff --git a/doc/api/oauth2.md b/doc/api/oauth2.md index be58d75333d..35c6eb4a982 100644 --- a/doc/api/oauth2.md +++ b/doc/api/oauth2.md @@ -15,16 +15,21 @@ To configure GitLab for this, see This functionality is based on the [doorkeeper Ruby gem](https://github.com/doorkeeper-gem/doorkeeper). -## CORS preflight requests +## Cross-origin resource sharing -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/364680) in GitLab 15.1. +> CORS preflight request support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/364680) in GitLab 15.1. -The following endpoints support [CORS preflight requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS): +Many `/oauth` endpoints support cross-origin resource sharing (CORS). From GitLab 15.1, the following endpoints also +support [CORS preflight requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS): - `/oauth/revoke` - `/oauth/token` - `/oauth/userinfo` +In addition to the headers listed for [simple requests](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS#simple_requests), +only the `Authorization` header can be used for preflight requests. For example, the `X-Requested-With` header +can't be used for preflight requests. + ## Supported OAuth 2.0 flows GitLab supports the following authorization flows: diff --git a/doc/api/openapi/openapi.yaml b/doc/api/openapi/openapi.yaml index cc6a161c783..a3a0485428d 100644 --- a/doc/api/openapi/openapi.yaml +++ b/doc/api/openapi/openapi.yaml @@ -1,5 +1,7 @@ openapi: 3.0.0 tags: + - name: metadata + description: Metadata of the GitLab instance - name: version description: Version - name: access_requests @@ -39,6 +41,10 @@ components: name: Private-Token paths: + # METADATA + /v4/metadata: + $ref: 'v4/metadata.yaml' + # VERSION /v4/version: $ref: 'v4/version.yaml' @@ -49,7 +55,7 @@ paths: /v4/projects/{id}/access_requests/{user_id}/approve: $ref: 'v4/access_requests.yaml#/accessRequestsProjectsApprove' - + /v4/projects/{id}/access_requests/{user_id}: $ref: 'v4/access_requests.yaml#/accessRequestsProjectsDeny' @@ -59,7 +65,7 @@ paths: /v4/groups/{id}/access_requests/{user_id}/approve: $ref: 'v4/access_requests.yaml#/accessRequestsGroupsApprove' - + /v4/groups/{id}/access_requests/{user_id}: $ref: 'v4/access_requests.yaml#/accessRequestsGroupsDeny' @@ -68,4 +74,4 @@ paths: $ref: 'v4/access_tokens.yaml#/accessTokens' /v4/projects/{id}/access_tokens/{token_id}: - $ref: 'v4/access_tokens.yaml#/accessTokensRevoke'
\ No newline at end of file + $ref: 'v4/access_tokens.yaml#/accessTokensRevoke' diff --git a/doc/api/openapi/v4/metadata.yaml b/doc/api/openapi/v4/metadata.yaml new file mode 100644 index 00000000000..6a5ef9f3355 --- /dev/null +++ b/doc/api/openapi/v4/metadata.yaml @@ -0,0 +1,43 @@ +# Markdown documentation: https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/api/metadata.md + +get: + tags: + - metadata + summary: "Retrieve metadata information for this GitLab instance." + operationId: "getMetadata" + responses: + "401": + description: "unauthorized operation" + "200": + description: "successful operation" + content: + "application/json": + schema: + title: "MetadataResponse" + type: "object" + properties: + version: + type: "string" + revision: + type: "string" + kas: + type: "object" + properties: + enabled: + type: "boolean" + externalUrl: + type: "string" + nullable: true + version: + type: "string" + nullable: true + examples: + Example: + value: + version: "15.0-pre" + revision: "c401a659d0c" + kas: + enabled: true + externalUrl: "grpc://gitlab.example.com:8150" + version: "15.0.0" + diff --git a/doc/api/packages/conan.md b/doc/api/packages/conan.md index 546a472ea53..1590893d006 100644 --- a/doc/api/packages/conan.md +++ b/doc/api/packages/conan.md @@ -19,6 +19,10 @@ NOTE: These endpoints do not adhere to the standard API authentication methods. See each route for details on how credentials are expected to be passed. +NOTE: +The Conan registry is not FIPS compliant and is disabled when [FIPS mode](../../development/fips_compliance.md) is enabled. +These endpoints will all return 404 Not Found. + ## Route prefix There are two sets of identical routes that each make requests in different scopes: diff --git a/doc/api/packages/debian.md b/doc/api/packages/debian.md index 66377850c49..4abb7bc7112 100644 --- a/doc/api/packages/debian.md +++ b/doc/api/packages/debian.md @@ -22,6 +22,10 @@ For instructions on how to upload and install Debian packages from the GitLab package registry, see the [Debian registry documentation](../../user/packages/debian_repository/index.md). NOTE: +The Debian registry is not FIPS compliant and is disabled when [FIPS mode](../../development/fips_compliance.md) is enabled. +These endpoints will all return `404 Not Found`. + +NOTE: These endpoints do not adhere to the standard API authentication methods. See the [Debian registry documentation](../../user/packages/debian_repository/index.md) for details on which headers and token types are supported. diff --git a/doc/api/packages/debian_group_distributions.md b/doc/api/packages/debian_group_distributions.md index 0a1b7b4571e..0d0a4cb2cde 100644 --- a/doc/api/packages/debian_group_distributions.md +++ b/doc/api/packages/debian_group_distributions.md @@ -18,6 +18,10 @@ This API is under development and is not meant for production use. For more information about working with Debian packages, see the [Debian package registry documentation](../../user/packages/debian_repository/index.md). +NOTE: +The Debian registry is not FIPS compliant and is disabled when [FIPS mode](../../development/fips_compliance.md) is enabled. +These endpoints will all return `404 Not Found`. + ## Enable the Debian group API Debian group repository support is still a work in progress. It's gated behind a feature flag that's diff --git a/doc/api/packages/debian_project_distributions.md b/doc/api/packages/debian_project_distributions.md index 533742642fd..4f3ac62f576 100644 --- a/doc/api/packages/debian_project_distributions.md +++ b/doc/api/packages/debian_project_distributions.md @@ -18,6 +18,10 @@ This API is under development and is not meant for production use. For more information about working with Debian packages, see the [Debian package registry documentation](../../user/packages/debian_repository/index.md). +NOTE: +The Debian registry is not FIPS compliant and is disabled when [FIPS mode](../../development/fips_compliance.md) is enabled. +These endpoints will all return `404 Not Found`. + ## Enable the Debian API The Debian API is behind a feature flag that is disabled by default. diff --git a/doc/api/packages/pypi.md b/doc/api/packages/pypi.md index 3e23ded61f4..e6204d87e1f 100644 --- a/doc/api/packages/pypi.md +++ b/doc/api/packages/pypi.md @@ -20,6 +20,10 @@ These endpoints do not adhere to the standard API authentication methods. See the [PyPI package registry documentation](../../user/packages/pypi_repository/index.md) for details on which headers and token types are supported. +NOTE: +[Twine 3.4.2](https://twine.readthedocs.io/en/stable/changelog.html?highlight=FIPS#id28) or greater +is recommended when [FIPS mode](../../development/fips_compliance.md) is enabled. + ## Download a package file from a group > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225545) in GitLab 13.12. diff --git a/doc/api/project_level_variables.md b/doc/api/project_level_variables.md index 4205e6699fe..81bb4a26614 100644 --- a/doc/api/project_level_variables.md +++ b/doc/api/project_level_variables.md @@ -170,5 +170,5 @@ This parameter is used for filtering by attributes, such as `environment_scope`. Example usage: ```shell -curl --request DELETE --globoff --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables/VARIABLE_1?filter[environment_scope]=production" +curl --globoff --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/1/variables/VARIABLE_1?filter[environment_scope]=production" ``` diff --git a/doc/api/project_repository_storage_moves.md b/doc/api/project_repository_storage_moves.md index ebb15e1c1d6..48c22ff2d93 100644 --- a/doc/api/project_repository_storage_moves.md +++ b/doc/api/project_repository_storage_moves.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Project repository storage moves API **(FREE SELF)** diff --git a/doc/api/projects.md b/doc/api/projects.md index 3cb75c39980..8a8fe522b63 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -15,7 +15,7 @@ The visibility level is determined by the `visibility` field in the project. Values for the project visibility level are: -- `private`: project access must be granted explicitly for each user. +- `private`: project access must be granted explicitly to each user. - `internal`: the project can be cloned by any signed-in user except [external users](../user/permissions.md#external-users). - `public`: the project can be accessed without any authentication. @@ -28,11 +28,10 @@ There are three options for `merge_method` to choose from: - `merge`: a merge commit is created for every merge, and merging is allowed if there are no conflicts. - `rebase_merge`: a merge commit is created for every merge, but merging is only - allowed if fast-forward merge is possible. This way you could make sure that - if this merge request would build, after merging to target branch it would - also build. -- `ff`: no merge commits are created and all merges are fast-forwarded, which - means that merging is only allowed if the branch could be fast-forwarded. + allowed if fast-forward merge is possible. You can make sure that the target + branch would build after this merge request builds and merges. +- `ff`: no merge commits are created and all merges are fast-forwarded. Merging + is only allowed if the branch could be fast-forwarded. ## List all projects @@ -68,6 +67,7 @@ GET /projects | `starred` | boolean | **{dotted-circle}** No | Limit by projects starred by the current user. | | `statistics` | boolean | **{dotted-circle}** No | Include project statistics. Only available to Reporter or higher level role members. | | `topic` | string | **{dotted-circle}** No | Comma-separated topic names. Limit results to projects that match all of given topics. See `topics` attribute. | +| `topic_id` | integer | **{dotted-circle}** No | Limit results to projects with the assigned topic given by the topic ID. | | `visibility` | string | **{dotted-circle}** No | Limit by visibility `public`, `internal`, or `private`. | | `wiki_checksum_failed` **(PREMIUM)** | boolean | **{dotted-circle}** No | Limit projects where the wiki checksum calculation has failed. | | `with_custom_attributes` | boolean | **{dotted-circle}** No | Include [custom attributes](custom_attributes.md) in response. _(administrator only)_ | @@ -80,17 +80,26 @@ for selected `order_by` options. When `simple=true` or the user is unauthenticated this returns something like: +Example request: + +```shell +curl --request GET "https://gitlab.example.com/api/v4/projects" +``` + +Example response: + ```json [ { "id": 4, "description": null, - "default_branch": "master", - "ssh_url_to_repo": "git@example.com:diaspora/diaspora-client.git", - "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", - "web_url": "http://example.com/diaspora/diaspora-client", - "readme_url": "http://example.com/diaspora/diaspora-client/blob/master/README.md", - "tag_list": [ //deprecated, use `topics` instead + "name": "Diaspora Client", + "name_with_namespace": "Diaspora / Diaspora Client", + "path": "diaspora-client", + "path_with_namespace": "diaspora/diaspora-client", + "created_at": "2013-09-30T13:46:02Z", + "default_branch": "main", + "tag_list": [ "example", "disapora client" ], @@ -98,21 +107,28 @@ When `simple=true` or the user is unauthenticated this returns something like: "example", "disapora client" ], - "name": "Diaspora Client", - "name_with_namespace": "Diaspora / Diaspora Client", - "path": "diaspora-client", - "path_with_namespace": "diaspora/diaspora-client", - "created_at": "2013-09-30T13:46:02Z", - "last_activity_at": "2013-09-30T13:46:02Z", + "ssh_url_to_repo": "git@gitlab.example.com:diaspora/diaspora-client.git", + "http_url_to_repo": "https://gitlab.example.com/diaspora/diaspora-client.git", + "web_url": "https://gitlab.example.com/diaspora/diaspora-client", + "readme_url": "https://gitlab.example.com/diaspora/diaspora-client/blob/master/README.md", + "avatar_url": "https://gitlab.example.com/uploads/project/avatar/4/uploads/avatar.png", "forks_count": 0, - "avatar_url": "http://example.com/uploads/project/avatar/4/uploads/avatar.png", - "star_count": 0 + "star_count": 0, + "last_activity_at": "2013-09-30T13:46:02Z", + "namespace": { + "id": 2, + "name": "Diaspora", + "path": "diaspora", + "kind": "group", + "full_path": "diaspora", + "parent_id": null, + "avatar_url": null, + "web_url": "https://gitlab.example.com/diaspora" + } }, { - "id": 6, - "description": null, - "default_branch": "master", -... + ... + } ``` When the user is authenticated and `simple` is not set this returns something like: @@ -122,12 +138,12 @@ When the user is authenticated and `simple` is not set this returns something li { "id": 4, "description": null, - "default_branch": "master", - "visibility": "private", - "ssh_url_to_repo": "git@example.com:diaspora/diaspora-client.git", - "http_url_to_repo": "http://example.com/diaspora/diaspora-client.git", - "web_url": "http://example.com/diaspora/diaspora-client", - "readme_url": "http://example.com/diaspora/diaspora-client/blob/master/README.md", + "name": "Diaspora Client", + "name_with_namespace": "Diaspora / Diaspora Client", + "path": "diaspora-client", + "path_with_namespace": "diaspora/diaspora-client", + "created_at": "2013-09-30T13:46:02Z", + "default_branch": "main", "tag_list": [ //deprecated, use `topics` instead "example", "disapora client" @@ -136,202 +152,115 @@ When the user is authenticated and `simple` is not set this returns something li "example", "disapora client" ], - "owner": { - "id": 3, - "name": "Diaspora", - "created_at": "2013-09-30T13:46:02Z" - }, - "name": "Diaspora Client", - "name_with_namespace": "Diaspora / Diaspora Client", - "path": "diaspora-client", - "path_with_namespace": "diaspora/diaspora-client", - "issues_enabled": true, - "open_issues_count": 1, - "merge_requests_enabled": true, - "jobs_enabled": true, - "wiki_enabled": true, - "snippets_enabled": false, - "can_create_merge_request_in": true, - "resolve_outdated_diff_discussions": false, - "container_registry_enabled": false, // deprecated, use container_registry_access_level instead - "container_registry_access_level": "disabled", - "security_and_compliance_access_level": "disabled", - "created_at": "2013-09-30T13:46:02Z", - "last_activity_at": "2013-09-30T13:46:02Z", - "creator_id": 3, + "ssh_url_to_repo": "git@gitlab.example.com:diaspora/diaspora-client.git", + "http_url_to_repo": "https://gitlab.example.com/diaspora/diaspora-client.git", + "web_url": "https://gitlab.example.com/diaspora/diaspora-client", + "readme_url": "https://gitlab.example.com/diaspora/diaspora-client/blob/master/README.md", + "avatar_url": "https://gitlab.example.com/uploads/project/avatar/4/uploads/avatar.png", + "forks_count": 0, + "star_count": 0, + "last_activity_at": "2022-06-24T17:11:26.841Z", "namespace": { "id": 3, "name": "Diaspora", "path": "diaspora", "kind": "group", - "full_path": "diaspora" - }, - "import_status": "none", - "archived": false, - "avatar_url": "http://example.com/uploads/project/avatar/4/uploads/avatar.png", - "shared_runners_enabled": true, - "forks_count": 0, - "star_count": 0, - "runners_token": "b8547b1dc37721d05889db52fa2f02", - "ci_default_git_depth": 50, - "ci_forward_deployment_enabled": true, - "public_jobs": true, - "shared_with_groups": [], - "only_allow_merge_if_pipeline_succeeds": false, - "allow_merge_on_skipped_pipeline": false, - "restrict_user_defined_variables": false, - "only_allow_merge_if_all_discussions_are_resolved": false, - "remove_source_branch_after_merge": false, - "request_access_enabled": false, - "merge_method": "merge", - "squash_option": "default_on", - "autoclose_referenced_issues": true, - "suggestion_commit_message": null, - "merge_commit_template": null, - "squash_commit_template": null, - "marked_for_deletion_at": "2020-04-03", // Deprecated and will be removed in API v5 in favor of marked_for_deletion_on - "marked_for_deletion_on": "2020-04-03", - "statistics": { - "commit_count": 37, - "storage_size": 1038090, - "repository_size": 1038090, - "wiki_size" : 0, - "lfs_objects_size": 0, - "job_artifacts_size": 0, - "pipeline_artifacts_size": 0, - "packages_size": 0, - "snippets_size": 0, - "uploads_size": 0 + "full_path": "diaspora", + "parent_id": null, + "avatar_url": "https://gitlab.example.com/uploads/project/avatar/6/uploads/avatar.png", + "web_url": "https://gitlab.example.com/diaspora" }, - "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-client", + "container_registry_image_prefix": "registry.gitlab.example.com/diaspora/diaspora-client", "_links": { - "self": "http://example.com/api/v4/projects", - "issues": "http://example.com/api/v4/projects/1/issues", - "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", - "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", - "labels": "http://example.com/api/v4/projects/1/labels", - "events": "http://example.com/api/v4/projects/1/events", - "members": "http://example.com/api/v4/projects/1/members", - "cluster_agents": "http://example.com/api/v4/projects/1/cluster_agents" - } - }, - { - "id": 6, - "description": null, - "default_branch": "master", - "visibility": "private", - "ssh_url_to_repo": "git@example.com:brightbox/puppet.git", - "http_url_to_repo": "http://example.com/brightbox/puppet.git", - "web_url": "http://example.com/brightbox/puppet", - "readme_url": "http://example.com/brightbox/puppet/blob/master/README.md", - "tag_list": [ //deprecated, use `topics` instead - "example", - "puppet" - ], - "topics": [ - "example", - "puppet" - ], - "owner": { - "id": 4, - "name": "Brightbox", - "created_at": "2013-09-30T13:46:02Z" + "self": "https://gitlab.example.com/api/v4/projects/4", + "issues": "https://gitlab.example.com/api/v4/projects/4/issues", + "merge_requests": "https://gitlab.example.com/api/v4/projects/4/merge_requests", + "repo_branches": "https://gitlab.example.com/api/v4/projects/4/repository/branches", + "labels": "https://gitlab.example.com/api/v4/projects/4/labels", + "events": "https://gitlab.example.com/api/v4/projects/4/events", + "members": "https://gitlab.example.com/api/v4/projects/4/members", + "cluster_agents": "https://gitlab.example.com/api/v4/projects/4/cluster_agents" + }, + "packages_enabled": true, + "empty_repo": false, + "archived": false, + "visibility": "public", + "resolve_outdated_diff_discussions": false, + "container_expiration_policy": { + "cadence": "1month", + "enabled": true, + "keep_n": 1, + "older_than": "14d", + "name_regex": "", + "name_regex_keep": ".*-main", + "next_run_at": "2022-06-25T17:11:26.865Z" }, - "name": "Puppet", - "name_with_namespace": "Brightbox / Puppet", - "path": "puppet", - "path_with_namespace": "brightbox/puppet", "issues_enabled": true, - "open_issues_count": 1, "merge_requests_enabled": true, - "jobs_enabled": true, "wiki_enabled": true, - "snippets_enabled": false, + "jobs_enabled": true, + "snippets_enabled": true, + "container_registry_enabled": true, + "service_desk_enabled": true, "can_create_merge_request_in": true, - "resolve_outdated_diff_discussions": false, - "container_registry_enabled": false, // deprecated, use container_registry_access_level instead - "container_registry_access_level": "disabled", - "security_and_compliance_access_level": "disabled", - "created_at": "2013-09-30T13:46:02Z", - "last_activity_at": "2013-09-30T13:46:02Z", - "creator_id": 3, - "namespace": { - "id": 4, - "name": "Brightbox", - "path": "brightbox", - "kind": "group", - "full_path": "brightbox" - }, - "import_status": "none", - "import_error": null, - "permissions": { - "project_access": { - "access_level": 10, - "notification_level": 3 - }, - "group_access": { - "access_level": 50, - "notification_level": 3 - } - }, - "archived": false, - "avatar_url": null, + "issues_access_level": "enabled", + "repository_access_level": "enabled", + "merge_requests_access_level": "enabled", + "forking_access_level": "enabled", + "wiki_access_level": "enabled", + "builds_access_level": "enabled", + "snippets_access_level": "enabled", + "pages_access_level": "enabled", + "operations_access_level": "enabled", + "analytics_access_level": "enabled", + "container_registry_access_level": "enabled", + "security_and_compliance_access_level": "private", + "emails_disabled": null, "shared_runners_enabled": true, - "forks_count": 0, - "star_count": 0, - "runners_token": "b8547b1dc37721d05889db52fa2f02", - "ci_default_git_depth": 0, + "lfs_enabled": true, + "creator_id": 1, + "import_status": "none", + "open_issues_count": 0, + "ci_default_git_depth": 20, "ci_forward_deployment_enabled": true, + "ci_job_token_scope_enabled": false, + "ci_separated_caches": true, "public_jobs": true, + "build_timeout": 3600, + "auto_cancel_pending_pipelines": "enabled", + "build_coverage_regex": null, + "ci_config_path": "", "shared_with_groups": [], "only_allow_merge_if_pipeline_succeeds": false, - "allow_merge_on_skipped_pipeline": false, + "allow_merge_on_skipped_pipeline": null, "restrict_user_defined_variables": false, + "request_access_enabled": true, "only_allow_merge_if_all_discussions_are_resolved": false, - "remove_source_branch_after_merge": false, - "request_access_enabled": false, + "remove_source_branch_after_merge": true, + "printing_merge_request_link_enabled": true, "merge_method": "merge", - "squash_option": "default_on", - "auto_devops_enabled": true, - "auto_devops_deploy_strategy": "continuous", - "repository_storage": "default", - "approvals_before_merge": 0, - "mirror": false, - "mirror_user_id": 45, - "mirror_trigger_builds": false, - "only_mirror_protected_branches": false, - "mirror_overwrites_diverged_branches": false, - "external_authorization_classification_label": null, - "packages_enabled": true, - "service_desk_enabled": false, - "service_desk_address": null, - "autoclose_referenced_issues": true, + "squash_option": "default_off", + "enforce_auth_checks_on_uploads": true, "suggestion_commit_message": null, "merge_commit_template": null, "squash_commit_template": null, - "statistics": { - "commit_count": 12, - "storage_size": 2066080, - "repository_size": 2066080, - "wiki_size" : 0, - "lfs_objects_size": 0, - "job_artifacts_size": 0, - "pipeline_artifacts_size": 0, - "packages_size": 0, - "snippets_size": 0, - "uploads_size": 0 - }, - "container_registry_image_prefix": "registry.example.com/brightbox/puppet", - "_links": { - "self": "http://example.com/api/v4/projects", - "issues": "http://example.com/api/v4/projects/1/issues", - "merge_requests": "http://example.com/api/v4/projects/1/merge_requests", - "repo_branches": "http://example.com/api/v4/projects/1/repository_branches", - "labels": "http://example.com/api/v4/projects/1/labels", - "events": "http://example.com/api/v4/projects/1/events", - "members": "http://example.com/api/v4/projects/1/members", - "cluster_agents": "http://example.com/api/v4/projects/1/cluster_agents" + "auto_devops_enabled": false, + "auto_devops_deploy_strategy": "continuous", + "autoclose_referenced_issues": true, + "keep_latest_artifact": true, + "runner_token_expiration_interval": null, + "external_authorization_classification_label": "", + "requirements_enabled": false, + "requirements_access_level": "enabled", + "security_and_compliance_enabled": false, + "compliance_frameworks": [], + "permissions": { + "project_access": null, + "group_access": null } + }, + { + ... } ] ``` @@ -365,6 +294,12 @@ You can filter by [custom attributes](custom_attributes.md) with: GET /projects?custom_attributes[key]=value&custom_attributes[other_key]=other_value ``` +Example request: + +```shell +curl --globoff --request GET "https://gitlab.example.com/api/v4/projects?custom_attributes[location]=Antarctica&custom_attributes[role]=Developer" +``` + ### Pagination limits In GitLab 13.0 and later, [offset-based pagination](index.md#offset-based-pagination) @@ -470,6 +405,7 @@ GET /users/:user_id/projects "runners_token": "b8547b1dc37721d05889db52fa2f02", "ci_default_git_depth": 50, "ci_forward_deployment_enabled": true, + "ci_separated_caches": true, "public_jobs": true, "shared_with_groups": [], "only_allow_merge_if_pipeline_succeeds": false, @@ -577,6 +513,7 @@ GET /users/:user_id/projects "runners_token": "b8547b1dc37721d05889db52fa2f02", "ci_default_git_depth": 0, "ci_forward_deployment_enabled": true, + "ci_separated_caches": true, "public_jobs": true, "shared_with_groups": [], "only_allow_merge_if_pipeline_succeeds": false, @@ -983,6 +920,7 @@ GET /projects/:id "runners_token": "b8bc4a7a29eb76ea83cf79e4908c2b", "ci_default_git_depth": 50, "ci_forward_deployment_enabled": true, + "ci_separated_caches": true, "public_jobs": true, "shared_with_groups": [ { @@ -1422,6 +1360,7 @@ Supported attributes: | `ci_config_path` | string | **{dotted-circle}** No | The path to CI configuration file. | | `ci_default_git_depth` | integer | **{dotted-circle}** No | Default number of revisions for [shallow cloning](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone). | | `ci_forward_deployment_enabled` | boolean | **{dotted-circle}** No | When a new deployment job starts, [skip older deployment jobs](../ci/pipelines/settings.md#skip-outdated-deployment-jobs) that are still pending | +| `ci_separated_caches` | boolean | **{dotted-circle}** No | Set whether or not caches should be [separated](../ci/caching/index.md#cache-key-names) by branch protection status. | | `container_expiration_policy_attributes` | hash | **{dotted-circle}** No | Update the image cleanup policy for this project. Accepts: `cadence` (string), `keep_n` (integer), `older_than` (string), `name_regex` (string), `name_regex_delete` (string), `name_regex_keep` (string), `enabled` (boolean). | | `container_registry_access_level` | string | **{dotted-circle}** No | Set visibility of container registry, for this project, to one of `disabled`, `private` or `enabled`. | | `container_registry_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable container registry for this project. Use `container_registry_access_level` instead. | @@ -1997,6 +1936,7 @@ Example response: "runners_token": "b8bc4a7a29eb76ea83cf79e4908c2b", "ci_default_git_depth": 50, "ci_forward_deployment_enabled": true, + "ci_separated_caches": true, "public_jobs": true, "shared_with_groups": [], "only_allow_merge_if_pipeline_succeeds": false, @@ -2122,6 +2062,7 @@ Example response: "runners_token": "b8bc4a7a29eb76ea83cf79e4908c2b", "ci_default_git_depth": 50, "ci_forward_deployment_enabled": true, + "ci_separated_caches": true, "public_jobs": true, "shared_with_groups": [], "only_allow_merge_if_pipeline_succeeds": false, @@ -2637,7 +2578,7 @@ PUT /projects/:id/push_rule ### Delete project push rule -> - Moved to GitLab Premium in 13.9. +> Moved to GitLab Premium in 13.9. Removes a push rule from a project. This is an idempotent method and can be called multiple times. Either the push rule is available or not. @@ -2654,7 +2595,7 @@ DELETE /projects/:id/push_rule > The `_links.cluster_agents` attribute in the response [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/347047) in GitLab 14.10. -See the [Project documentation](../user/project/settings/index.md#transferring-an-existing-project-into-another-namespace) +See the [Project documentation](../user/project/settings/index.md#transfer-a-project-to-another-namespace) for prerequisites to transfer a project. ```plaintext @@ -2820,7 +2761,7 @@ with the API scope enabled. ## Start the pull mirroring process for a Project **(PREMIUM)** -> - Moved to GitLab Premium in 13.9. +> Moved to GitLab Premium in 13.9. ```plaintext POST /projects/:id/mirror/pull diff --git a/doc/api/protected_tags.md b/doc/api/protected_tags.md index e2b27692373..b2ab3227a7e 100644 --- a/doc/api/protected_tags.md +++ b/doc/api/protected_tags.md @@ -100,6 +100,7 @@ curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitla | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) owned by the authenticated user | | `name` | string | yes | The name of the tag or wildcard | | `create_access_level` | string | no | Access levels allowed to create (defaults: `40`, Maintainer role) | +| `allowed_to_create` | array | no | Array of access levels allowed to create tags, with each described by a hash of the form `{user_id: integer}`, `{group_id: integer}`, or `{access_level: integer}` | Example response: diff --git a/doc/api/repositories.md b/doc/api/repositories.md index 1a1eee175f7..34b86902271 100644 --- a/doc/api/repositories.md +++ b/doc/api/repositories.md @@ -319,6 +319,7 @@ Supported attributes: | `date` | datetime | no | The date and time of the release, defaults to the current time. | | `branch` | string | no | The branch to commit the changelog changes to, defaults to the project's default branch. | | `trailer` | string | no | The Git trailer to use for including commits, defaults to `Changelog`. | +| `config_file` | string | no | The path of changelog configuration file in the project's Git repository, defaults to `.gitlab/changelog_config.yml`. | | `file` | string | no | The file to commit the changes to, defaults to `CHANGELOG.md`. | | `message` | string | no | The commit message to produce when committing the changes, defaults to `Add changelog for version X` where X is the value of the `version` argument. | @@ -453,7 +454,7 @@ If a revert commit includes the trailer used for generating changelogs ### Customize the changelog output The output is customized using a YAML configuration file stored in your -project's Git repository. This file must reside in +project's Git repository. This default configuration file path is `.gitlab/changelog_config.yml`. You can set the following variables in this file: @@ -736,6 +737,7 @@ Supported attributes: | `to` | string | no | The end of the range of commits (as a SHA) to use for the changelog. This commit _is_ included in the list. Defaults to the branch specified in the `branch` attribute. | | `date` | datetime | no | The date and time of the release, ISO 8601 formatted. Example: `2016-03-11T03:45:40Z`. Defaults to the current time. | | `trailer` | string | no | The Git trailer to use for including commits, defaults to `Changelog`. | +| `config_file` | string | no | The path of changelog configuration file in the project's Git repository, defaults to `.gitlab/changelog_config.yml`. | ```shell curl --header "PRIVATE-TOKEN: token" "https://gitlab.com/api/v4/projects/42/repository/changelog?version=1.0.0" diff --git a/doc/api/settings.md b/doc/api/settings.md index 805797792de..28a33e4deb4 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -175,6 +175,7 @@ Example response: "container_registry_expiration_policies_caching": true, "container_registry_expiration_policies_worker_capacity": 4, "container_registry_token_expire_delay": 5, + "package_registry_cleanup_policies_worker_capacity": 2, "repository_storages": ["default"], "plantuml_enabled": false, "plantuml_url": null, @@ -271,6 +272,7 @@ listed in the descriptions of the relevant settings. | `container_registry_expiration_policies_caching` | boolean | no | Caching during the execution of [cleanup policies](../user/packages/container_registry/reduce_container_registry_storage.md#set-cleanup-limits-to-conserve-resources). | | `container_registry_expiration_policies_worker_capacity` | integer | no | Number of workers for [cleanup policies](../user/packages/container_registry/reduce_container_registry_storage.md#set-cleanup-limits-to-conserve-resources). | | `container_registry_token_expire_delay` | integer | no | Container Registry token duration in minutes. | +| `package_registry_cleanup_policies_worker_capacity` | integer | no | Number of workers assigned to the packages cleanup policies. | | `deactivate_dormant_users` | boolean | no | Enable [automatic deactivation of dormant users](../user/admin_area/moderate_users.md#automatically-deactivate-dormant-users). | | `default_artifacts_expire_in` | string | no | Set the default expiration time for each job's artifacts. | | `default_branch_name` | string | no | [Instance-level custom initial branch name](../user/project/repository/branches/default.md#instance-level-custom-initial-branch-name) ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225258) in GitLab 13.2). | @@ -379,6 +381,9 @@ listed in the descriptions of the relevant settings. | `max_personal_access_token_lifetime` **(ULTIMATE SELF)** | integer | no | Maximum allowable lifetime for access tokens in days. | | `max_ssh_key_lifetime` **(ULTIMATE SELF)** | integer | no | Maximum allowable lifetime for SSH keys in days. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1007) in GitLab 14.6. | | `metrics_method_call_threshold` | integer | no | A method call is only tracked when it takes longer than the given amount of milliseconds. | +| `max_number_of_repository_downloads` **(ULTIMATE SELF)** | integer | no | Maximum number of unique repositories a user can download in the specified time period before they are banned. Default: 0, Maximum: 10,000 repositories. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87980) in GitLab 15.1. | +| `max_number_of_repository_downloads_within_time_period` **(ULTIMATE SELF)** | integer | no | Reporting time period (in seconds). Default: 0, Maximum: 864000 seconds (10 days). [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87980) in GitLab 15.1. | +| `git_rate_limit_users_allowlist` **(ULTIMATE SELF)** | array of strings | no | List of usernames excluded from Git anti-abuse rate limits. Default: `[]`, Maximum: 100 usernames. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90815) in GitLab 15.2. | | `mirror_available` | boolean | no | Allow repository mirroring to configured by project Maintainers. If disabled, only Administrators can configure repository mirroring. | | `mirror_capacity_threshold` **(PREMIUM)** | integer | no | Minimum capacity to be available before scheduling more mirrors preemptively. | | `mirror_max_capacity` **(PREMIUM)** | integer | no | Maximum number of mirrors that can be synchronizing at the same time. | diff --git a/doc/api/snippets.md b/doc/api/snippets.md index 52bcd072de9..e3bc3573ed7 100644 --- a/doc/api/snippets.md +++ b/doc/api/snippets.md @@ -31,6 +31,15 @@ Get a list of the current user's snippets. GET /snippets ``` +Parameters: + +| Attribute | Type | Required | Description | +|------------------|----------|----------|-----------------------------------------------------------------------------------------------------| +| `per_page` | integer | no | Number of snippets to return per page. | +| `page` | integer | no | Page to retrieve. | +| `created_after` | datetime | no | Return snippets created after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | +| `created_before` | datetime | no | Return snippets created before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | + Example request: ```shell @@ -389,10 +398,12 @@ GET /snippets/public Parameters: -| Attribute | Type | Required | Description | -|:-----------|:--------|:---------|:---------------------------------------| -| `per_page` | integer | no | Number of snippets to return per page. | -| `page` | integer | no | Page to retrieve. | +| Attribute | Type | Required | Description | +|------------------|----------|----------|-----------------------------------------------------------------------------------------------------| +| `per_page` | integer | no | Number of snippets to return per page. | +| `page` | integer | no | Page to retrieve. | +| `created_after` | datetime | no | Return snippets created after the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | +| `created_before` | datetime | no | Return snippets created before the given time. Expected in ISO 8601 format (`2019-03-15T08:00:00Z`) | Example request: diff --git a/doc/api/usage_data.md b/doc/api/usage_data.md index be816a0f864..a01ed3e83a5 100644 --- a/doc/api/usage_data.md +++ b/doc/api/usage_data.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, api @@ -34,7 +34,7 @@ Example response: by month product_section: enablement product_stage: enablement - product_group: group::global search + product_group: global_search product_category: global_search value_type: number status: active @@ -79,6 +79,7 @@ Example response: "active_user_count": "SELECT COUNT(\"users\".\"id\") FROM \"users\" WHERE (\"users\".\"state\" IN ('active')) AND (\"users\".\"user_type\" IS NULL OR \"users\".\"user_type\" IN (NULL, 6, 4))", "edition": "EE", "license_md5": "c701acc03844c45366dd175ef7a4e19c", + "license_sha256": "366dd175ef7a4e19cc701acc03844c45366dd175ef7a4e19cc701acc03844c45", "license_id": null, "historical_max_users": 0, "licensee": { @@ -138,6 +139,7 @@ Sample response: "active_user_count": -3, "edition": "EE", "license_md5": "bb8cd0d8a6d9569ff3f70b8927a1f949", + "license_sha256": "366dd175ef7a4e19cc701acc03844c45366dd175ef7a4e19cc701acc03844c45", "license_id": null, "historical_max_users": 0, "licensee": { diff --git a/doc/api/users.md b/doc/api/users.md index 5e41a0f6258..06c0fadbbbf 100644 --- a/doc/api/users.md +++ b/doc/api/users.md @@ -57,8 +57,7 @@ NOTE: Username search is case insensitive. In addition, you can filter users based on the states `blocked` and `active`. -It does not support `active=false` or `blocked=false`. The list of billable users -is the total number of users minus the blocked users. +It does not support `active=false` or `blocked=false`. ```plaintext GET /users?active=true @@ -105,7 +104,7 @@ parameter `without_project_bots=true`. GET /users?without_project_bots=true ``` -### For administrators +### For administrators **(FREE SELF)** > The `namespace_id` field in the response was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82045) in GitLab 14.10. @@ -284,7 +283,7 @@ Parameters: | Attribute | Type | Required | Description | |-----------|---------|----------|------------------| -| `id` | integer | yes | The ID of a user | +| `id` | integer | yes | ID of a user | ```json { @@ -314,7 +313,7 @@ Parameters: } ``` -### For administrator +### For administrators **(FREE SELF)** > The `namespace_id` field in the response was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82045) in GitLab 14.10. @@ -326,7 +325,7 @@ Parameters: | Attribute | Type | Required | Description | |-----------|---------|----------|------------------| -| `id` | integer | yes | The ID of a user | +| `id` | integer | yes | ID of a user | Example Responses: @@ -477,7 +476,7 @@ Parameters: | `shared_runners_minutes_limit` **(PREMIUM)** | No | Can be set by administrators only. Maximum number of monthly CI/CD minutes for this user. Can be `nil` (default; inherit system default), `0` (unlimited), or `> 0`. | | `skip_confirmation` | No | Skip confirmation - true or false (default) | | `skype` | No | Skype ID | -| `theme_id` | No | The GitLab theme for the user (see [the user preference docs](../user/profile/preferences.md#navigation-theme) for more information) | +| `theme_id` | No | GitLab theme for the user (see [the user preference docs](../user/profile/preferences.md#navigation-theme) for more information) | | `twitter` | No | Twitter account | | `username` | Yes | Username | | `view_diffs_file_by_file` | No | Flag indicating the user sees only one file diff per page | @@ -507,7 +506,7 @@ Parameters: | `external` | No | Flags the user as external - true or false (default) | | `extra_shared_runners_minutes_limit` **(PREMIUM)** | No | Can be set by administrators only. Additional CI/CD minutes for this user. | | `group_id_for_saml` | No | ID of group where SAML has been configured | -| `id` | Yes | The ID of the user | +| `id` | Yes | ID of the user | | `linkedin` | No | LinkedIn | | `location` | No | User's location | | `name` | No | Name | @@ -517,11 +516,11 @@ Parameters: | `private_profile` | No | User's profile is private - true, false (default), or null (is converted to false) | | `projects_limit` | No | Limit projects each user can create | | `provider` | No | External provider name | -| `public_email` | No | The public email of the user (must be already verified) | +| `public_email` | No | Public email of the user (must be already verified) | | `shared_runners_minutes_limit` **(PREMIUM)** | No | Can be set by administrators only. Maximum number of monthly CI/CD minutes for this user. Can be `nil` (default; inherit system default), `0` (unlimited) or `> 0`. | | `skip_reconfirmation` | No | Skip reconfirmation - true or false (default) | | `skype` | No | Skype ID | -| `theme_id` | No | The GitLab theme for the user (see [the user preference docs](../user/profile/preferences.md#navigation-theme) for more information) | +| `theme_id` | No | GitLab theme for the user (see [the user preference docs](../user/profile/preferences.md#navigation-theme) for more information) | | `twitter` | No | Twitter account | | `username` | No | Username | | `view_diffs_file_by_file` | No | Flag indicating the user sees only one file diff per page | @@ -544,7 +543,7 @@ Parameters: | Attribute | Type | Required | Description | |------------|---------|----------|------------------------| -| `id` | integer | yes | The ID of a user | +| `id` | integer | yes | ID of a user | | `provider` | string | yes | External provider name | ## User deletion @@ -560,10 +559,14 @@ Parameters: | Attribute | Type | Required | Description | |---------------|---------|----------|----------------------------------------------| -| `id` | integer | yes | The ID of a user | +| `id` | integer | yes | ID of a user | | `hard_delete` | boolean | no | If true, contributions that would usually be [moved to Ghost User](../user/profile/account/delete_account.md#associated-records) are deleted instead, as well as groups owned solely by this user. | -## List current user (for normal users) +## List current user + +Get current user. + +### For normal users Gets currently authenticated user. @@ -619,7 +622,7 @@ GET /user Users on [GitLab Premium or higher](https://about.gitlab.com/pricing/) also see the `shared_runners_minutes_limit`, `extra_shared_runners_minutes_limit` parameters. -## List current user (for administrators) +### For administrators **(FREE SELF)** > The `namespace_id` field in the response was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82045) in GitLab 14.10. @@ -631,7 +634,7 @@ Parameters: | Attribute | Type | Required | Description | |-----------|---------|----------|--------------------------------------------------| -| `sudo` | integer | no | the ID of a user to make the call in their place | +| `sudo` | integer | no | ID of a user to make the call in their place | ```json { @@ -696,7 +699,7 @@ GET /user/status ``` ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/user/status" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/user/status" ``` Example response: @@ -721,7 +724,7 @@ GET /users/:id_or_username/status | Attribute | Type | Required | Description | | ---------------- | ------ | -------- | ------------------------------------------------- | -| `id_or_username` | string | yes | The ID or username of the user to get a status of | +| `id_or_username` | string | yes | ID or username of the user to get a status of | ```shell curl "https://gitlab.example.com/users/janedoe/status" @@ -749,8 +752,8 @@ PUT /user/status | Attribute | Type | Required | Description | | -------------------- | ------ | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| `emoji` | string | no | The name of the emoji to use as status. If omitted `speech_balloon` is used. Emoji name can be one of the specified names in the [Gemojione index](https://github.com/bonusly/gemojione/blob/master/config/index.json). | -| `message` | string | no | The message to set as a status. It can also contain emoji codes. | +| `emoji` | string | no | Name of the emoji to use as status. If omitted `speech_balloon` is used. Emoji name can be one of the specified names in the [Gemojione index](https://github.com/bonusly/gemojione/blob/master/config/index.json). | +| `message` | string | no | Message to set as a status. It can also contain emoji codes. | | `clear_status_after` | string | no | Automatically clean up the status after a given time interval, allowed values: `30_minutes`, `3_hours`, `8_hours`, `1_day`, `3_days`, `7_days`, `30_days` When both parameters `emoji` and `message` are empty, the status is cleared. When the `clear_status_after` parameter is missing from the request, the previously set value for `"clear_status_after` is cleared. @@ -818,7 +821,7 @@ Parameters: | `view_diffs_file_by_file` | Yes | Flag indicating the user sees only one file diff per page. | | `show_whitespace_in_diffs` | Yes | Flag indicating the user sees whitespace changes in diffs. | -## User Follow +## User follow ### Follow and unfollow users @@ -836,7 +839,7 @@ POST /users/:id/unfollow | Attribute | Type | Required | Description | | --------- | ------- | -------- | ---------------------------- | -| `id` | integer | yes | The ID of the user to follow | +| `id` | integer | yes | ID of the user to follow | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/users/3/follow" @@ -871,7 +874,7 @@ GET /users/:id/following | Attribute | Type | Required | Description | | --------- | ------- | -------- | ---------------------------- | -| `id` | integer | yes | The ID of the user to follow | +| `id` | integer | yes | ID of the user to follow | ```shell curl --request GET --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/users/3/followers" @@ -975,7 +978,7 @@ GET /users/:id_or_username/keys | Attribute | Type | Required | Description | | ---------------- | ------ | -------- | ------------------------------------------------------- | -| `id_or_username` | string | yes | The ID or username of the user to get the SSH keys for. | +| `id_or_username` | string | yes | ID or username of the user to get the SSH keys for. | ## Single SSH key @@ -989,7 +992,7 @@ Parameters: | Attribute | Type | Required | Description | |-----------|--------|----------|----------------------| -| `key_id` | string | yes | The ID of an SSH key | +| `key_id` | string | yes | ID of an SSH key | ```json { @@ -1038,9 +1041,9 @@ Parameters: | Attribute | Type | Required | Description | |--------------|--------|----------|--------------------------------------------------------------------------------| -| `title` | string | yes | new SSH key's title | -| `key` | string | yes | new SSH key | -| `expires_at` | string | no | The expiration date of the SSH key in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) | +| `title` | string | yes | New SSH key's title | +| `key` | string | yes | New SSH key | +| `expires_at` | string | no | Expiration date of the SSH key in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) | ```json { @@ -1079,9 +1082,9 @@ Parameters: | Attribute | Type | Required | Description | |--------------|---------|----------|--------------------------------------------------------------------------------| | `id` | integer | yes | ID of specified user | -| `title` | string | yes | new SSH key's title | -| `key` | string | yes | new SSH key | -| `expires_at` | string | no | The expiration date of the SSH key in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) | +| `title` | string | yes | New SSH key's title | +| `key` | string | yes | New SSH key | +| `expires_at` | string | no | Expiration date of the SSH key in ISO 8601 format (`YYYY-MM-DDTHH:MM:SSZ`) | NOTE: This also adds an audit event, as described in [audit instance events](../administration/audit_events.md#instance-events). **(PREMIUM)** @@ -1152,7 +1155,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `key_id` | integer | yes | The ID of the GPG key | +| `key_id` | integer | yes | ID of the GPG key | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/user/gpg_keys/1" @@ -1180,7 +1183,7 @@ Parameters: | Attribute | Type | Required | Description | |-----------|--------|----------|-----------------| -| `key` | string | yes | The new GPG key | +| `key` | string | yes | New GPG key | ```shell curl --data "key=-----BEGIN PGP PUBLIC KEY BLOCK-----\r\n\r\nxsBNBFV..." \ @@ -1211,7 +1214,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `key_id` | integer | yes | The ID of the GPG key | +| `key_id` | integer | yes | ID of the GPG key | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/user/gpg_keys/1" @@ -1231,7 +1234,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ------- | -------- | ------------------ | -| `id` | integer | yes | The ID of the user | +| `id` | integer | yes | ID of the user | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/2/gpg_keys" @@ -1262,8 +1265,8 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `id` | integer | yes | The ID of the user | -| `key_id` | integer | yes | The ID of the GPG key | +| `id` | integer | yes | ID of the user | +| `key_id` | integer | yes | ID of the GPG key | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/2/gpg_keys/1" @@ -1291,8 +1294,8 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `id` | integer | yes | The ID of the user | -| `key_id` | integer | yes | The ID of the GPG key | +| `id` | integer | yes | ID of the user | +| `key_id` | integer | yes | ID of the GPG key | ```shell curl --data "key=-----BEGIN PGP PUBLIC KEY BLOCK-----\r\n\r\nxsBNBFV..." \ @@ -1323,8 +1326,8 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ------- | -------- | --------------------- | -| `id` | integer | yes | The ID of the user | -| `key_id` | integer | yes | The ID of the GPG key | +| `id` | integer | yes | ID of the user | +| `key_id` | integer | yes | ID of the GPG key | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/2/gpg_keys/1" @@ -1627,8 +1630,8 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ------- | -------- | ---------------------------------------------------------- | -| `user_id` | integer | yes | The ID of the user | -| `state` | string | no | filter tokens based on state (`all`, `active`, `inactive`) | +| `user_id` | integer | yes | ID of the user | +| `state` | string | no | Filter tokens based on state (`all`, `active`, `inactive`) | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/42/impersonation_tokens" @@ -1761,8 +1764,8 @@ Parameters: | Attribute | Type | Required | Description | | ------------------------ | ------- | -------- | --------------------------------- | -| `user_id` | integer | yes | The ID of the user | -| `impersonation_token_id` | integer | yes | The ID of the impersonation token | +| `user_id` | integer | yes | ID of the user | +| `impersonation_token_id` | integer | yes | ID of the impersonation token | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/42/impersonation_tokens/2" @@ -1802,10 +1805,10 @@ POST /users/:user_id/impersonation_tokens | Attribute | Type | Required | Description | | ------------ | ------- | -------- | --------------------------------------------------------------------------- | -| `user_id` | integer | yes | The ID of the user | -| `name` | string | yes | The name of the impersonation token | -| `expires_at` | date | no | The expiration date of the impersonation token in ISO format (`YYYY-MM-DD`) | -| `scopes` | array | yes | The array of scopes of the impersonation token (`api`, `read_user`) | +| `user_id` | integer | yes | ID of the user | +| `name` | string | yes | Name of the impersonation token | +| `expires_at` | date | no | Expiration date of the impersonation token in ISO format (`YYYY-MM-DD`) | +| `scopes` | array | yes | Array of scopes of the impersonation token (`api`, `read_user`) | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "name=mytoken" --data "expires_at=2017-04-04" \ @@ -1845,8 +1848,8 @@ Parameters: | Attribute | Type | Required | Description | | ------------------------ | ------- | -------- | --------------------------------- | -| `user_id` | integer | yes | The ID of the user | -| `impersonation_token_id` | integer | yes | The ID of the impersonation token | +| `user_id` | integer | yes | ID of the user | +| `impersonation_token_id` | integer | yes | ID of the impersonation token | ```shell curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/42/impersonation_tokens/1" @@ -1867,10 +1870,10 @@ POST /users/:user_id/personal_access_tokens | Attribute | Type | Required | Description | | ------------ | ------- | -------- | ------------------------------------------------------------------------------------------------------------------------ | -| `user_id` | integer | yes | The ID of the user | -| `name` | string | yes | The name of the personal access token | -| `expires_at` | date | no | The expiration date of the personal access token in ISO format (`YYYY-MM-DD`) | -| `scopes` | array | yes | The array of scopes of the personal access token. See [personal access token scopes](../user/profile/personal_access_tokens.md#personal-access-token-scopes) for possible values. | +| `user_id` | integer | yes | ID of the user | +| `name` | string | yes | Name of the personal access token | +| `expires_at` | date | no | Expiration date of the personal access token in ISO format (`YYYY-MM-DD`) | +| `scopes` | array | yes | Array of scopes of the personal access token. See [personal access token scopes](../user/profile/personal_access_tokens.md#personal-access-token-scopes) for possible values. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" --data "name=mytoken" --data "expires_at=2017-04-04" \ @@ -1895,10 +1898,11 @@ Example response: } ``` -## Get user activities (administrator only) +## Get user activities -NOTE: -This API endpoint is only available on 8.15 (EE) and 9.1 (CE) and above. +Pre-requisite: + +- You must be an administrator. Get the last activity date for all users, sorted from oldest to newest. @@ -1921,7 +1925,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ------ | -------- | ---------------------------------------------------------------------------------------------- | -| `from` | string | no | Date string in the format YEAR-MONTH-DAY. For example, `2016-03-11`. Defaults to 6 months ago. | +| `from` | string | no | Date string in the format `YEAR-MM-DD`. For example, `2016-03-11`. Defaults to 6 months ago. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/user/activities" @@ -1951,11 +1955,15 @@ Example response: `last_activity_at` is deprecated. Use `last_activity_on` instead. -## User memberships (administrator only) +## User memberships > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20532) in GitLab 12.8. -Lists all projects and groups a user is a member of. This endpoint is available for administrators only. +Pre-requisite: + +- You must be an administrator. + +Lists all projects and groups a user is a member of. It returns the `source_id`, `source_name`, `source_type` and `access_level` of a membership. Source can be of type `Namespace` (representing a group) or `Project`. The response represents only direct memberships. Inherited memberships, for example in subgroups, are not included. Access levels are represented by an integer value. For more details, read about the meaning of [access level values](access_requests.md#valid-access-levels). @@ -1968,7 +1976,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ------- | -------- | ------------------------------------------------------------------ | -| `id` | integer | yes | The ID of a specified user | +| `id` | integer | yes | ID of a specified user | | `type` | string | no | Filter memberships by type. Can be either `Project` or `Namespace` | Returns: @@ -2000,3 +2008,37 @@ Example response: } ] ``` + +## Disable two factor authentication + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/295260) in GitLab 15.2. + +Pre-requisite: + +- You must be an administrator. + +Disables two factor authentication (2FA) for the specified user. + +Administrators cannot disable 2FA for their own user account or other administrators using the API. Instead, they can disable an +administrator's 2FA [using the Rails console](../security/two_factor_authentication.md#for-a-single-user). + +```plaintext +PATCH /users/:id/disable_two_factor +``` + +Parameters: + +| Attribute | Type | Required | Description | +| --------- | ------- | -------- | --------------------- | +| `id` | integer | yes | ID of the user | + +```shell +curl --request PATCH --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/users/1/disable_two_factor" +``` + +Returns: + +- `204 No content` on success. +- `400 Bad request` if two factor authentication is not enabled for the specified user. +- `403 Forbidden` if not authenticated as an administrator. +- `404 User Not Found` if user cannot be found. diff --git a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md index 40b29ebb6ea..c536ff59b84 100644 --- a/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md +++ b/doc/ci/ci_cd_for_external_repos/bitbucket_integration.md @@ -18,8 +18,13 @@ To use GitLab CI/CD with a Bitbucket Cloud repository: 1. On the top menu, select **Projects > Create new project**. 1. Select **Run CI/CD for external repository**. 1. Select **Repository by URL**. + 1. Fill in the fields with information from the repository in Bitbucket: + - For **Git repository URL**, use the URL from the **Clone this repository** panel in Bitbucket. + - Leave the username blank. + - You can generate and use a [Bitbucket App Password](https://support.atlassian.com/bitbucket-cloud/docs/app-passwords/) for the password field. GitLab imports the repository and enables [Pull Mirroring](../../user/project/repository/mirror/pull.md). + You can check that mirroring is working in the project by going to **Settings > Repository > Mirroring repositories**. 1. In GitLab, create a [Personal Access Token](../../user/profile/personal_access_tokens.md) diff --git a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md index 9af5218e058..aea7b492d4e 100644 --- a/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md +++ b/doc/ci/cloud_deployment/ecs/deploy_to_aws_ecs.md @@ -223,7 +223,7 @@ These variables are injected into the pipeline jobs and can access the ECS API. |`AWS_SECRET_ACCESS_KEY`|`<Secret access key of the deployer>`| For authenticating `aws` CLI. | |`AWS_DEFAULT_REGION`|`us-east-2`| For authenticating `aws` CLI. | |`CI_AWS_ECS_CLUSTER`|`ecs-demo`| The ECS cluster is accessed by `production_ecs` job. | - |`CI_AWS_ECS_SERVICE`|`ecs_demo`| The ECS service of the cluster is updated by `production_ecs` job. | + |`CI_AWS_ECS_SERVICE`|`ecs_demo`| The ECS service of the cluster is updated by `production_ecs` job. Ensure that this variable is scoped to the appropriate environment (`production`, `staging`, `review/*`). | |`CI_AWS_ECS_TASK_DEFINITION`|`ecs_demo`| The ECS task definition is updated by `production_ecs` job. | ### Make a change to the demo application @@ -246,6 +246,55 @@ NOTE: ECS deploy jobs wait for the rollout to complete before exiting. To disable this behavior, set `CI_AWS_ECS_WAIT_FOR_ROLLOUT_COMPLETE_DISABLED` to a non-empty value. +## Set up Review Apps + +To use [Review Apps](../../../development/testing_guide/review_apps.md) with ECS: + +1. Set up a new [service](#create-an-ecs-service). +1. Use the `CI_AWS_ECS_SERVICE` variable to set the name. +1. Set the environment scope to `review/*`. + +Only one Review App at a time can be deployed because this service is shared by all review apps. + +## Set up Security Testing + +### Configure SAST + +To use [SAST](../../../user/application_security/sast/index.md) with ECS, add the following to your `.gitlab-ci.yml` file: + +```yaml +include: + - template: Security/SAST.gitlab-ci.yml +``` + +For more details and configuration options, see the [SAST documentation](../../../user/application_security/sast/index.md#configuration). + +### Configure DAST + +To use [DAST](../../../user/application_security/dast/index.md) on non-default branches, [set up review apps](#set-up-review-apps) +and add the following to your `.gitlab-ci.yml` file: + +```yaml +include: + - template: Security/DAST.gitlab-ci.yml +``` + +To use DAST on the default branch: + +1. Set up a new [service](#create-an-ecs-service). This service will be used to deploy a temporary +DAST environment. +1. Use the `CI_AWS_ECS_SERVICE` variable to set the name. +1. Set the scope to the `dast-default` environment. +1. Add the following to your `.gitlab-ci.yml` file: + +```yaml +include: + - template: Security/DAST.gitlab-ci.yml + - template: Jobs/DAST-Default-Branch-Deploy.gitlab-ci.yml +``` + +For more details and configuration options, see the [DAST documentation](../../../user/application_security/dast/index.md). + ## Further reading - If you're interested in more of the continuous deployments to clouds, see [cloud deployments](../index.md). diff --git a/doc/ci/cloud_deployment/heroku.md b/doc/ci/cloud_deployment/heroku.md new file mode 100644 index 00000000000..4e627675b01 --- /dev/null +++ b/doc/ci/cloud_deployment/heroku.md @@ -0,0 +1,32 @@ +--- +stage: Release +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Use GitLab CI/CD to deploy to Heroku + +You can deploy an application to Heroku by using GitLab CI/CD. + +## Prerequisites + +- A [Heroku](https://id.heroku.com/login) account. + Sign in with an existing Heroku account or create a new one. + +## Deploy to Heroku + +1. In Heroku: + 1. Create an application and copy the application name. + 1. Browse to **Account Settings** and copy the API key. +1. In your GitLab project, create two [variables](../../ci/variables/index.md): + - `HEROKU_APP_NAME` for the application name. + - `HEROKU_PRODUCTION_KEY` for the API key +1. Edit your `.gitlab-ci.yml` file to add the Heroku deployment command. This example uses the `dpl` gem for Ruby: + + ```yaml + heroku_deploy: + stage: production + script: + - gem install dpl + - dpl --provider=heroku --app=$HEROKU_APP_NAME --api-key=$HEROKU_PRODUCTION_KEY + ``` diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index c5be2328264..5df396e796e 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -93,7 +93,7 @@ To deploy to your ECS cluster: | Environment variable name | Value | |:-------------------------------|:------------------------| | `CI_AWS_ECS_CLUSTER` | The name of the AWS ECS cluster that you're targeting for your deployments. | - | `CI_AWS_ECS_SERVICE` | The name of the targeted service tied to your AWS ECS cluster. | + | `CI_AWS_ECS_SERVICE` | The name of the targeted service tied to your AWS ECS cluster. Ensure that this variable is scoped to the appropriate environment (`production`, `staging`, `review/*`). | | `CI_AWS_ECS_TASK_DEFINITION` | If the task definition is in ECS, the name of the task definition tied to the service. | | `CI_AWS_ECS_TASK_DEFINITION_FILE` | If the task definition is a JSON file in GitLab, the filename, including the path. For example, `ci/aws/my_task_definition.json`. If the name of the task definition in your JSON file is the same name as an existing task definition in ECS, then a new revision is created when CI/CD runs. Otherwise, a brand new task definition is created, starting at revision 1. | diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index b13bd041d46..6ffa68e4873 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -406,8 +406,11 @@ When you stop an environment: to the list of **Stopped** environments. - An [`on_stop` action](../yaml/index.md#environmenton_stop), if defined, is executed. -Dynamic environments stop automatically when their associated branch is -deleted. +There are multiple ways to clean up [dynamic environments](#create-a-dynamic-environment): + +- If you use [merge request pipelines](../pipelines/merge_request_pipelines.md), GitLab stops an environment [when a merge request is merged or closed](#stop-an-environment-when-a-merge-request-is-merged-or-closed). +- If you do _NOT_ use [merge request pipelines](../pipelines/merge_request_pipelines.md), GitLab stops an environment [when the associated feature branch is deleted](#stop-an-environment-when-a-branch-is-deleted). +- If you set [an expiry period to an environment](../yaml/index.md#environmentauto_stop_in), GitLab stops an environment [when it's expired](#stop-an-environment-after-a-certain-time-period). #### Stop an environment when a branch is deleted @@ -425,8 +428,6 @@ deploy_review: name: review/$CI_COMMIT_REF_SLUG url: https://$CI_ENVIRONMENT_SLUG.example.com on_stop: stop_review - rules: - - if: $CI_MERGE_REQUEST_ID stop_review: stage: deploy @@ -435,9 +436,7 @@ stop_review: environment: name: review/$CI_COMMIT_REF_SLUG action: stop - rules: - - if: $CI_MERGE_REQUEST_ID - when: manual + when: manual ``` Both jobs must have the same [`rules`](../yaml/index.md#rules) @@ -455,6 +454,39 @@ try to check out the code after the branch is deleted. Read more in the [`.gitlab-ci.yml` reference](../yaml/index.md#environmenton_stop). +#### Stop an environment when a merge request is merged or closed + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/60885) in GitLab 11.10. + +You can configure environments to stop when a merge request is merged or closed. +This stop trigger is automatically enabled when you use [merge request pipelines](../pipelines/merge_request_pipelines.md). + +The following example shows a `deploy_review` job that calls a `stop_review` job +to clean up and stop the environment. + +```yaml +deploy_review: + stage: deploy + script: + - echo "Deploy a review app" + environment: + name: review/$CI_COMMIT_REF_SLUG + on_stop: stop_review + rules: + - if: $CI_MERGE_REQUEST_ID + +stop_review: + stage: deploy + script: + - echo "Remove review app" + environment: + name: review/$CI_COMMIT_REF_SLUG + action: stop + rules: + - if: $CI_MERGE_REQUEST_ID + when: manual +``` + #### Stop an environment when another job is finished You can set an environment to stop when another job is finished. @@ -642,6 +674,18 @@ To delete a stopped environment in the GitLab UI: 1. Next to the environment you want to delete, select **Delete environment**. 1. On the confirmation dialog box, select **Delete environment**. +#### Delete an active environment without running a stop job + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225794) in GitLab 15.1. + +You can delete an active environment without running a stop job. +This is useful when you have an active environment, but the corresponding `action: stop` job can't run or succeed for some reason. + +To delete an active environment: + +1. Execute the [Stop an environment API](../../api/environments.md#stop-an-environment) while specifying `force=true`. +1. Execute the [Delete an environment API](../../api/environments.md#delete-an-environment). + ### Access an environment for preparation or verification purposes > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208655) in GitLab 13.2. diff --git a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md index 389429f3f0f..90cbcb9e240 100644 --- a/doc/ci/examples/authenticating-with-hashicorp-vault/index.md +++ b/doc/ci/examples/authenticating-with-hashicorp-vault/index.md @@ -33,29 +33,30 @@ Each job has JSON Web Token (JWT) provided as CI/CD variable named `CI_JOB_JWT`. The following fields are included in the JWT: -| Field | When | Description | -| ----------------------- | ------ | ----------- | -| `jti` | Always | Unique identifier for this token | -| `iss` | Always | Issuer, the domain of your GitLab instance | -| `iat` | Always | Issued at | -| `nbf` | Always | Not valid before | -| `exp` | Always | Expires at | -| `sub` | Always | Subject (job ID) | -| `namespace_id` | Always | Use this to scope to group or user level namespace by ID | -| `namespace_path` | Always | Use this to scope to group or user level namespace by path | -| `project_id` | Always | Use this to scope to project by ID | -| `project_path` | Always | Use this to scope to project by path | -| `user_id` | Always | ID of the user executing the job | -| `user_login` | Always | Username of the user executing the job | -| `user_email` | Always | Email of the user executing the job | -| `pipeline_id` | Always | ID of this pipeline | -| `pipeline_source` | Always | [Pipeline source](../../jobs/job_control.md#common-if-clauses-for-rules) | -| `job_id` | Always | ID of this job | -| `ref` | Always | Git ref for this job | -| `ref_type` | Always | Git ref type, either `branch` or `tag` | -| `ref_protected` | Always | `true` if this Git ref is protected, `false` otherwise | -| `environment` | Job is creating a deployment | Environment this job deploys to ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294440) in GitLab 13.9) | -| `environment_protected` | Job is creating a deployment |`true` if deployed environment is protected, `false` otherwise ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294440) in GitLab 13.9) | +| Field | When | Description | +|-------------------------|------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `jti` | Always | Unique identifier for this token | +| `iss` | Always | Issuer, the domain of your GitLab instance | +| `iat` | Always | Issued at | +| `nbf` | Always | Not valid before | +| `exp` | Always | Expires at | +| `sub` | Always | Subject (job ID) | +| `namespace_id` | Always | Use this to scope to group or user level namespace by ID | +| `namespace_path` | Always | Use this to scope to group or user level namespace by path | +| `project_id` | Always | Use this to scope to project by ID | +| `project_path` | Always | Use this to scope to project by path | +| `user_id` | Always | ID of the user executing the job | +| `user_login` | Always | Username of the user executing the job | +| `user_email` | Always | Email of the user executing the job | +| `pipeline_id` | Always | ID of this pipeline | +| `pipeline_source` | Always | [Pipeline source](../../jobs/job_control.md#common-if-clauses-for-rules) | +| `job_id` | Always | ID of this job | +| `ref` | Always | Git ref for this job | +| `ref_type` | Always | Git ref type, either `branch` or `tag` | +| `ref_protected` | Always | `true` if this Git ref is protected, `false` otherwise | +| `environment` | Job specifies an environment | Environment this job specifies ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294440) in GitLab 13.9) | +| `environment_protected` | Job specifies an environment | `true` if specified environment is protected, `false` otherwise ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294440) in GitLab 13.9) | +| `deployment_tier` | Job specifies an environment | [Deployment tier](../../environments/index.md#deployment-tier-of-environments) of environment this job specifies ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363590) in GitLab 15.2) | Example JWT payload: diff --git a/doc/ci/examples/index.md b/doc/ci/examples/index.md index 0fc7b06a584..77591ad2ca6 100644 --- a/doc/ci/examples/index.md +++ b/doc/ci/examples/index.md @@ -24,11 +24,11 @@ The following table lists examples with step-by-step tutorials that are containe | Use case | Resource | |-------------------------------|----------| -| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](../../user/project/merge_requests/browser_performance_testing.md). | +| Browser performance testing | [Browser Performance Testing with the Sitespeed.io container](../testing/browser_performance_testing.md). | | Deployment with Dpl | [Using `dpl` as deployment tool](deployment/index.md). | | GitLab Pages | See the [GitLab Pages](../../user/project/pages/index.md) documentation for a complete example of deploying a static site. | | End-to-end testing | [End-to-end testing with GitLab CI/CD and WebdriverIO](end_to_end_testing_webdriverio/index.md). | -| Load performance testing | [Load Performance Testing with the k6 container](../../user/project/merge_requests/load_performance_testing.md). | +| Load performance testing | [Load Performance Testing with the k6 container](../testing/load_performance_testing.md). | | Multi project pipeline | [Build, test deploy using multi project pipeline](https://gitlab.com/gitlab-examples/upstream-project). | | npm with semantic-release | [Publish npm packages to the GitLab Package Registry using semantic-release](semantic-release.md). | | PHP with Laravel, Envoy | [Test and deploy Laravel applications with GitLab CI/CD and Envoy](laravel_with_gitlab_and_envoy/index.md). | @@ -95,6 +95,7 @@ choose one of these templates: - [Rust (`Rust.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Rust.gitlab-ci.yml) - [Scala (`Scala.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Scala.gitlab-ci.yml) - [Swift (`Swift.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Swift.gitlab-ci.yml) +- [Terraform (`Terraform.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.gitlab-ci.yml) - [Terraform (`Terraform.latest.gitlab-ci.yml`)](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Terraform.latest.gitlab-ci.yml) If a programming language or framework template is not in this list, you can contribute diff --git a/doc/ci/index.md b/doc/ci/index.md index 4fe2dee32bf..80752830b85 100644 --- a/doc/ci/index.md +++ b/doc/ci/index.md @@ -83,8 +83,8 @@ GitLab CI/CD features, grouped by DevOps stage, include: | [ChatOps](chatops/index.md) | Trigger CI jobs from chat, with results sent back to the channel. | | [Connect to cloud services](cloud_services/index.md) | Connect to cloud providers using OpenID Connect (OIDC) to retrieve temporary credentials to access services or secrets. | | **Verify** | | -| [Browser Performance Testing](../user/project/merge_requests/browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | -| [Load Performance Testing](../user/project/merge_requests/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | +| [Browser Performance Testing](testing/browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | +| [Load Performance Testing](testing/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | | [CI services](services/index.md) | Link Docker containers with your base image. | | [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | | [Interactive Web Terminals](interactive_web_terminal/index.md) | Open an interactive web terminal to debug the running jobs. | @@ -101,7 +101,7 @@ GitLab CI/CD features, grouped by DevOps stage, include: | [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | | [Cloud deployment](cloud_deployment/index.md) | Deploy your application to a main cloud provider. | | **Secure** | | -| [Code Quality](../user/project/merge_requests/code_quality.md) | Analyze your source code quality. | +| [Code Quality](testing/code_quality.md) | Analyze your source code quality. | | [Container Scanning](../user/application_security/container_scanning/index.md) | Check your Docker containers for known vulnerabilities. | | [Dependency Scanning](../user/application_security/dependency_scanning/index.md) | Analyze your dependencies for known vulnerabilities. | | [License Compliance](../user/compliance/license_compliance/index.md) | Search your project dependencies for their licenses. | diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index 1f7a7436c0a..fd6afb1a0ad 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -45,9 +45,6 @@ Clicking an individual job shows you its job log, and allows you to: ### View all jobs in a project -> - An improved view was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293862) in GitLab 14.10, [with a flag](../../administration/feature_flags.md) named `jobs_table_vue`. Disabled by default. -> - The job status filter was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/82539) in GitLab 14.10, [with a flag](../../administration/feature_flags.md) named `jobs_table_vue_search`. Disabled by default. - To view the full list of jobs that ran in a project: 1. On the top bar, select **Menu > Projects** and find the project. @@ -136,9 +133,9 @@ jobs. Select to expand them. To create a group of jobs, in the [CI/CD pipeline configuration file](../yaml/index.md), separate each job name with a number and one of the following: -- A slash (`/`), for example, `test 1/3`, `test 2/3`, `test 3/3`. -- A colon (`:`), for example, `test 1:3`, `test 2:3`, `test 3:3`. -- A space, for example `test 0 3`, `test 1 3`, `test 2 3`. +- A slash (`/`), for example, `slash-test 1/3`, `slash-test 2/3`, `slash-test 3/3`. +- A colon (`:`), for example, `colon-test 1:3`, `colon-test 2:3`, `colon-test 3:3`. +- A space, for example `space-test 0 3`, `space-test 1 3`, `space-test 2 3`. You can use these symbols interchangeably. @@ -358,7 +355,7 @@ Add `[collapsed=true]` after the section name and before the `\r`. The section e remains unchanged: - Section start marker with `[collapsed=true]`: `\e[0Ksection_start:UNIX_TIMESTAMP:SECTION_NAME[collapsed=true]\r\e[0K` + `TEXT_OF_SECTION_HEADER` -- Section end marker: `section_end:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` +- Section end marker: `\e[0Ksection_end:UNIX_TIMESTAMP:SECTION_NAME\r\e[0K` Add the updated section start text to the CI configuration. For example, using `echo`: diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index 83747f36597..24133fe9a9b 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -547,25 +547,29 @@ You can use [protected branches](../../user/project/protected_branches.md) to mo ### Types of manual jobs -Manual jobs can be either optional or blocking: - -- **Optional**: The default setting for manual jobs. - - They have [`allow_failure: true`](../yaml/index.md#allow_failure) by default. - - The status does not contribute to the overall pipeline status. A pipeline can - succeed even if all of its manual jobs fail. -- **Blocking**: An optional setting for manual jobs. - - Add `allow_failure: false` to the job configuration. - - The pipeline stops at the stage where the job is defined. To let the pipeline - continue running, [run the manual job](#run-a-manual-job). - - Merge requests in projects with [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md) - enabled can't be merged with a blocked pipeline. Blocked pipelines show a status - of **blocked**. +Manual jobs can be either optional or blocking. -### Run a manual job +In optional manual jobs: + +- [`allow_failure`](../yaml/index.md#allow_failure) is `true`, which is the default + setting for jobs that have `when: manual` and no [`rules`](../yaml/index.md#rules), + or `when: manual` defined outside of `rules`. +- The status does not contribute to the overall pipeline status. A pipeline can + succeed even if all of its manual jobs fail. + +In blocking manual jobs: -To run a manual job, you must have permission to merge to the assigned branch. +- `allow_failure` is `false`, which is the default setting for jobs that have `when: manual` + defined inside [`rules`](../yaml/index.md#rules). +- The pipeline stops at the stage where the job is defined. To let the pipeline + continue running, [run the manual job](#run-a-manual-job). +- Merge requests in projects with [merge when pipeline succeeds](../../user/project/merge_requests/merge_when_pipeline_succeeds.md) + enabled can't be merged with a blocked pipeline. +- The pipeline shows a status of **blocked**. + +### Run a manual job -To run a manual job: +To run a manual job, you must have permission to merge to the assigned branch: 1. Go to the pipeline, job, [environment](../environments/index.md#configure-manual-deployments), or deployment view. diff --git a/doc/ci/metrics_reports.md b/doc/ci/metrics_reports.md index 03869a639f1..c3ab3392f36 100644 --- a/doc/ci/metrics_reports.md +++ b/doc/ci/metrics_reports.md @@ -1,68 +1,11 @@ --- -stage: Verify -group: Pipeline Insights -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'testing/metrics_reports.md' +remove_date: '2022-08-31' --- -# Metrics Reports **(PREMIUM)** +This document was moved to [another location](testing/metrics_reports.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9788) in GitLab 11.10. Requires GitLab Runner 11.10 and above. - -GitLab provides a lot of great reporting tools for things like [merge requests](../user/project/merge_requests/index.md) - [Unit test reports](testing/unit_test_reports.md), [code quality](../user/project/merge_requests/code_quality.md), and performance tests. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. - -You can configure your job to use custom Metrics Reports, and GitLab displays a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. - -![Metrics Reports](img/metrics_reports_v13_0.png) - -## Use cases - -Consider the following examples of data that can use Metrics Reports: - -1. Memory usage -1. Load testing results -1. Code complexity -1. Code coverage stats - -## How it works - -Metrics for a branch are read from the latest metrics report artifact (default filename: `metrics.txt`) as string values. - -For an MR, the values of these metrics from the feature branch are compared to the values from the target branch. Then they are displayed in the MR widget in this order: - -- Existing metrics with changed values. -- Metrics that have been added by the MR. Marked with a **New** badge. -- Metrics that have been removed by the MR. Marked with a **Removed** badge. -- Existing metrics with unchanged values. - -## How to set it up - -Add a job that creates a [metrics report](yaml/artifacts_reports.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. - -For example: - -```yaml -metrics: - script: - - echo 'metric_name metric_value' > metrics.txt - artifacts: - reports: - metrics: metrics.txt -``` - -## Advanced Example - -An advanced example of an OpenMetrics text file (from the [Prometheus documentation](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-format-example)) -renders in the merge request widget as: - -![Metrics Reports Advanced](img/metrics_reports_advanced_v13_0.png) - -## Troubleshooting - -### Metrics reports did not change - -You can see `Metrics reports did not change` when trying to view metrics reports in merge requests. Reasons for this are: - -- The target branch for the merge request doesn't have a baseline metrics report for comparison. -- You don't have GitLab license at the Premium tier or above. - -There is [an issue open](https://gitlab.com/gitlab-org/gitlab/-/issues/343065) to improve this message. +<!-- This redirect file can be deleted after <2022-09-22>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/migration/circleci.md b/doc/ci/migration/circleci.md index b1c4c62c465..3b890458e56 100644 --- a/doc/ci/migration/circleci.md +++ b/doc/ci/migration/circleci.md @@ -289,8 +289,8 @@ Self-managed runners: GitLab.com shared runners: - Linux -- Windows -- [Planned: macOS](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/5720) +- [Windows](../runners/saas/windows_saas_runner.md) ([Beta](../../policy/alpha-beta-support.md#beta-features)). +- [macOS](../runners/saas/macos_saas_runner.md) ([Beta](../../policy/alpha-beta-support.md#beta-features)). ### Machine and specific build environments diff --git a/doc/ci/pipeline_editor/index.md b/doc/ci/pipeline_editor/index.md index d87b336224c..4fb2ec94d60 100644 --- a/doc/ci/pipeline_editor/index.md +++ b/doc/ci/pipeline_editor/index.md @@ -98,7 +98,7 @@ where: - [YAML `!reference` tags](../yaml/yaml_optimization.md#reference-tags) are also replaced with the linked configuration. -Using `!refence` tags can cause nested configuration that display with +Using `!reference` tags can cause nested configuration that display with multiple hyphens (`-`) in the expanded view. This behavior is expected, and the extra hyphens do not affect the job's execution. For example, this configuration and fully expanded version are both valid: diff --git a/doc/ci/pipelines/cicd_minutes.md b/doc/ci/pipelines/cicd_minutes.md index ff30d5701cc..1e862c87035 100644 --- a/doc/ci/pipelines/cicd_minutes.md +++ b/doc/ci/pipelines/cicd_minutes.md @@ -7,22 +7,33 @@ type: reference # CI/CD minutes quota **(PREMIUM)** -[Shared runners](../runners/runners_scope.md#shared-runners) are shared with every project and group in a GitLab instance. -When jobs run on shared runners, CI/CD minutes are used. +Administrators can limit the amount of time that projects can use to run jobs on +[shared runners](../runners/runners_scope.md#shared-runners) each month. This limit +is tracked with a quota of CI/CD minutes. -You can set limits on the number of CI/CD minutes that are used each month. +By default, one minute of execution time by a single job uses +one CI/CD minute. The total amount of CI/CD minutes used by a pipeline is +[the sum of all its jobs' durations](#how-cicd-minute-usage-is-calculated). +Jobs can run concurrently, so the total CI/CD minute usage can be higher than the +end-to-end duration of a pipeline. -- On GitLab.com, the quota of CI/CD minutes is set for each [namespace](../../user/group/index.md#namespaces), - and is determined by [your license tier](https://about.gitlab.com/pricing/). -- On self-managed GitLab instances, the quota of CI/CD minutes for each namespace is set by administrators. +On GitLab.com: -In addition to the monthly quota, you can add more CI/CD minutes when needed. +- CI/CD minutes quotas are enabled for both public and private projects, but public + projects [consume CI/CD minutes at a slower rate](#cost-factor). +- The base monthly CI/CD minutes quota for a GitLab.com [namespace](../../user/group/index.md#namespaces) + is determined by its [license tier](https://about.gitlab.com/pricing/). +- You can [purchase additional CI/CD minutes](#purchase-additional-cicd-minutes) + if you need more than the number of CI/CD minutes in your monthly quota. -- On GitLab.com, you can [purchase additional CI/CD minutes](#purchase-additional-cicd-minutes). -- On self-managed GitLab instances, administrators can [assign more CI/CD minutes](#set-the-quota-of-cicd-minutes-for-a-specific-namespace). +On self-managed GitLab instances: -[Specific runners](../runners/runners_scope.md#specific-runners) -are not subject to a quota of CI/CD minutes. +- CI/CD minutes quotas are disabled by default. +- When enabled, CI/CD minutes quotas apply to private projects only. +- Administrators can [assign more CI/CD minutes](#set-the-quota-of-cicd-minutes-for-a-specific-namespace) + if a namespace uses all the CI/CD minutes in its monthly quota. + +[Specific runners](../runners/runners_scope.md#specific-runners) are not subject to a quota of CI/CD minutes. ## Set the quota of CI/CD minutes for all namespaces @@ -160,46 +171,43 @@ To purchase additional minutes for your personal namespace: After your payment is processed, the additional CI/CD minutes are added to your personal namespace. -## How CI/CD minutes are calculated - -CI/CD minutes for individual jobs are calculated based on: +## How CI/CD minute usage is calculated -- The duration the job runs. -- The visibility of the projects where the job runs. - -GitLab uses this formula to calculate CI/CD minutes consumed by a job: +GitLab uses this formula to calculate the CI/CD minute usage of a job: ```plaintext Job duration * Cost factor ``` -- **Job duration**: The time, in seconds, that a job took to run on a shared runner. - It does not include time spent in `created` or `pending` status. -- **Cost factor**: A number based on project visibility. +- **Job duration**: The time, in seconds, that a job took to run on a shared runner, + not including time spent in the `created` or `pending` statuses. +- [**Cost factor**](#cost-factor): A number based on project visibility. -The number is transformed into minutes and added to the overall quota in the job's top-level namespace. +The value is transformed into minutes and added to the count of used CI/CD minutes +in the job's top-level namespace. -For example: +For example, if a user `alice` runs a pipeline: -- A user, `alice`, runs a pipeline under the `gitlab-org` namespace. -- The CI/CD minutes consumed by each job in the pipeline are added to the - overall consumption for the `gitlab-org` namespace, not the `alice` namespace. -- If a pipeline runs for one of the personal projects for `alice`, the CI/CD minutes - are added to the overall consumption for the `alice` namespace. +- Under the `gitlab-org` namespace, the CI/CD minutes used by each job in the pipeline are + added to the overall consumption for the `gitlab-org` namespace, not the `alice` namespace. +- For one of the personal projects in their namespace, the CI/CD minutes are added + to the overall consumption for the `alice` namespace. The CI/CD minutes used by one pipeline is the total CI/CD minutes used by all the jobs -that ran in the pipeline. The CI/CD minute usage for a pipeline can be higher than -the duration of the pipeline if many jobs ran at the same time. +that ran in the pipeline. Jobs can run concurrently, so the total CI/CD minutes usage +can be higher than the end-to-end duration of a pipeline. ### Cost factor -The cost factor for a job running on a shared runner is: +The cost factors for jobs running on shared runners on GitLab.com are: + +- `0.008` for public projects, and projects in the [GitLab for Open Source program](../../subscriptions/index.md#gitlab-for-open-source). + For every 125 minutes of job execution time, you use 1 CI/CD minute. +- `1` for internal and private projects. + +The cost factors on self-managed instances are: -- `0.008` for public projects on GitLab SaaS, if [created 2021-07-17 or later](https://gitlab.com/gitlab-org/gitlab/-/issues/332708). - (For every 125 minutes of job time, you accrue 1 CI/CD minute.) -- `0.008` for projects members of GitLab [Open Source program](../../subscriptions/index.md#gitlab-for-open-source). - (For every 125 minutes of job time, you accrue 1 CI/CD minute.) -- `0` for public projects on GitLab self-managed instances, and for GitLab SaaS public projects created before 2021-07-17. +- `0` for public projects, so they do not consume CI/CD minutes. - `1` for internal and private projects. ### Additional costs on GitLab SaaS diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 76419e61661..08264170d52 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -40,7 +40,7 @@ A typical pipeline might consist of four stages, executed in the following order NOTE: If you have a [mirrored repository that GitLab pulls from](../../user/project/repository/mirror/pull.md), you may need to enable pipeline triggering in your project's -**Settings > Repository > Pull from a remote repository > Trigger pipelines for mirror updates**. +**Settings > Repository > Mirroring repositories > Trigger pipelines for mirror updates**. ## Types of pipelines @@ -374,7 +374,7 @@ GitLab capitalizes the stages' names in the pipeline graphs. ### View full pipeline graph -> - Visualization improvements [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276949) in GitLab 13.11. +> Visualization improvements [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276949) in GitLab 13.11. The [pipeline details page](#view-pipelines) displays the full pipeline graph of all the jobs in the pipeline. diff --git a/doc/ci/pipelines/job_artifacts.md b/doc/ci/pipelines/job_artifacts.md index b0336d0499f..c8babe3320d 100644 --- a/doc/ci/pipelines/job_artifacts.md +++ b/doc/ci/pipelines/job_artifacts.md @@ -430,3 +430,24 @@ successful_test_job: paths: - bin/results ``` + +### Error message `FATAL: invalid argument` when uploading a dotenv artifact on a windows runner + +The PowerShell `echo` command writes files with UCS-2 LE BOM (Byte Order Mark) encoding, +but only UTF-8 is supported. If you try create the dotenv artifact with `echo`, it causes a +`FATAL: invalid argument` error. + +Use PowerShell `Add-Content` instead, which uses UTF-8: + +```yaml +test-job: + stage: test + tags: + - windows + script: + - echo "test job" + - Add-Content -Path build.env -Value "MY_ENV_VAR=true" + artifacts: + reports: + dotenv: build.env +``` diff --git a/doc/ci/pipelines/merge_trains.md b/doc/ci/pipelines/merge_trains.md index e0fe59d1ab8..ac644628f3a 100644 --- a/doc/ci/pipelines/merge_trains.md +++ b/doc/ci/pipelines/merge_trains.md @@ -69,7 +69,6 @@ To enable merge trains: - You must have the Maintainer role. - You must be using [GitLab Runner](https://gitlab.com/gitlab-org/gitlab-runner) 11.9 or later. -- In GitLab 13.0 and later, you need [Redis](https://redis.io/) 5.0 or later. - Your repository must be a GitLab repository, not an [external repository](../ci_cd_for_external_repos/index.md). diff --git a/doc/ci/pipelines/merged_results_pipelines.md b/doc/ci/pipelines/merged_results_pipelines.md index 9661d2f5263..777871a7c5f 100644 --- a/doc/ci/pipelines/merged_results_pipelines.md +++ b/doc/ci/pipelines/merged_results_pipelines.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Merged results pipelines **(PREMIUM)** -> [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/351192) from `pipelines for merged results` to `merged results pipelines` in GitLab 14.8. +> - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/351192) from `pipelines for merged results` to `merged results pipelines` in GitLab 14.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91849) in GitLab 15.1, merged results pipelines also run on [Draft merge requests](../../user/project/merge_requests/drafts.md). A *merged results pipeline* is a type of [merge request pipeline](merge_request_pipelines.md). It is a pipeline that runs against the results of the source and target branches merged together. @@ -19,10 +20,7 @@ The pipeline runs against the target branch as it exists at the moment you run t Over time, while you're working in the source branch, the target branch might change. Any time you want to be sure the merged results are accurate, you should re-run the pipeline. -Merged results pipelines can't run when: - -- The target branch has changes that conflict with the changes in the source branch. -- The merge request is a [**Draft** merge request](../../user/project/merge_requests/drafts.md). +Merged results pipelines can't run when the target branch has changes that conflict with the changes in the source branch. In these cases, the pipeline runs as a [merge request pipeline](merge_request_pipelines.md) and [is labeled as `merge request`](merge_request_pipelines.md#types-of-merge-request-pipelines). diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md index 20184635e4a..dfbd2708d8d 100644 --- a/doc/ci/pipelines/multi_project_pipelines.md +++ b/doc/ci/pipelines/multi_project_pipelines.md @@ -188,7 +188,7 @@ Upstream pipelines take precedence over downstream ones. If there are two variables with the same name defined in both upstream and downstream projects, the ones defined in the upstream project take precedence. -#### Pass CI/CD variables to a downstream pipeline by using variable inheritance +#### Pass CI/CD variables to a downstream pipeline by using variable inheritance **(PREMIUM)** You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](../variables/index.md#pass-an-environment-variable-to-another-job) and [`needs:project`](../yaml/index.md#needsproject). diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md index 3a1367f4a8c..07e1c8a6d99 100644 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ b/doc/ci/pipelines/pipeline_artifacts.md @@ -10,7 +10,7 @@ Pipeline artifacts are files created by GitLab after a pipeline finishes. Pipeli different to [job artifacts](job_artifacts.md) because they are not explicitly managed by `.gitlab-ci.yml` definitions. -Pipeline artifacts are used by the [test coverage visualization feature](../../user/project/merge_requests/test_coverage_visualization.md) +Pipeline artifacts are used by the [test coverage visualization feature](../testing/test_coverage_visualization.md) to collect coverage information. ## Storage diff --git a/doc/ci/pipelines/pipeline_efficiency.md b/doc/ci/pipelines/pipeline_efficiency.md index 991b3aef76c..ad43895d7ef 100644 --- a/doc/ci/pipelines/pipeline_efficiency.md +++ b/doc/ci/pipelines/pipeline_efficiency.md @@ -75,7 +75,7 @@ The [Pipeline success and duration charts](index.md#pipeline-success-and-duratio give information about pipeline runtime and failed job counts. Tests like [unit tests](../testing/unit_test_reports.md), integration tests, end-to-end tests, -[code quality](../../user/project/merge_requests/code_quality.md) tests, and others +[code quality](../testing/code_quality.md) tests, and others ensure that problems are automatically found by the CI/CD pipeline. There could be many pipeline stages involved causing long runtimes. @@ -174,7 +174,7 @@ to stop them from running: Ensure that errors are detected early in the CI/CD pipeline. A job that takes a very long time to complete keeps a pipeline from returning a failed status until the job completes. -Design pipelines so that jobs that can [fail fast](../../user/project/merge_requests/fail_fast_testing.md) +Design pipelines so that jobs that can [fail fast](../testing/fail_fast_testing.md) run earlier. For example, add an early stage and move the syntax, style linting, Git commit message verification, and similar jobs in there. diff --git a/doc/ci/pipelines/pipelines_for_merged_results.md b/doc/ci/pipelines/pipelines_for_merged_results.md deleted file mode 100644 index 0c3100a51f1..00000000000 --- a/doc/ci/pipelines/pipelines_for_merged_results.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'merged_results_pipelines.md' -remove_date: '2022-04-27' ---- - -This document was moved to [another location](merged_results_pipelines.md). - -<!-- This redirect file can be deleted after 2022-04-27. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index 40daf154a5f..43f20bfa9ea 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -21,7 +21,7 @@ For public and internal projects, you can change who can see your: - Pipelines - Job output logs - Job artifacts -- [Pipeline security dashboard](../../user/application_security/security_dashboard/index.md#view-vulnerabilities-in-a-pipeline) +- [Pipeline security dashboard](../../user/application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline) To change the visibility of your pipelines and related features: @@ -288,6 +288,9 @@ regular expression displayed in the settings. Available in GitLab 14.10 and earl The regular expression you need is in the **Test coverage parsing** field. +If you need to retrieve the project coverage setting from many projects, you can +[use the API to programmatically retrieve the setting](https://gitlab.com/gitlab-org/gitlab/-/issues/17633#note_945941397). + <!-- end_remove --> ### Test coverage examples diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 3ebd38df8db..6dd03033926 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -287,6 +287,9 @@ can supply the ID by either: - Dynamically adding the `data-merge-request-id` value during the build of the app. - Supplying it manually through the visual review form in the app. +If the ID is missing from the `script`, the visual review tool prompts you to enter the +merge request ID before you can provide feedback. + ### Authentication for Visual Reviews > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/42750#note_317271120) in GitLab 12.10. diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md index 729de4d99d3..038bda4ab09 100644 --- a/doc/ci/runners/index.md +++ b/doc/ci/runners/index.md @@ -12,7 +12,7 @@ No configuration is required. Your jobs can run on: - [Linux runners](saas/linux_saas_runner.md). - [Windows runners](saas/windows_saas_runner.md) ([Beta](../../policy/alpha-beta-support.md#beta-features)). -- [macOS runners](saas/macos_saas_runner.md) ([Limited Availability](../../policy/alpha-beta-support.md#limited-availability-la)). +- [macOS runners](saas/macos_saas_runner.md) ([Beta](../../policy/alpha-beta-support.md#beta-features)). The number of minutes you can use on these runners depends on the [maximum number of CI/CD minutes](../pipelines/cicd_minutes.md) diff --git a/doc/ci/runners/saas/macos_saas_runner.md b/doc/ci/runners/saas/macos_saas_runner.md index 65404c89f9a..5a2d84b6996 100644 --- a/doc/ci/runners/saas/macos_saas_runner.md +++ b/doc/ci/runners/saas/macos_saas_runner.md @@ -4,9 +4,9 @@ group: Runner info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# SaaS runners on macOS (Limited Availability) **(PREMIUM SAAS)** +# SaaS runners on macOS (Beta) **(PREMIUM SAAS)** -SaaS runners on macOS are in [Limited Availability](../../../policy/alpha-beta-support.md#limited-availability-la) for approved open source programs and customers in Premium and Ultimate plans. +SaaS runners on macOS are in [Beta](../../../policy/alpha-beta-support.md#beta-features) for approved open source programs and customers in Premium and Ultimate plans. SaaS runners on macOS provide an on-demand macOS build environment integrated with GitLab SaaS [CI/CD](../../../ci/index.md). @@ -18,13 +18,15 @@ CI/CD minutes used on GitLab SaaS macOS runners are included in your CI/CD minut Refer to the CI/CD minutes [cost factor](../../../ci/pipelines/cicd_minutes.md#cost-factor) for the cost factor applied to the GitLab SaaS macOS runners. +Jobs handled by macOS shared runners on GitLab.com **time out after 2 hours**, regardless of the timeout configured in a project. + ## Access request process -While in limited availability, to run CI jobs on the macOS runners, GitLab SaaS customer namespaces must be explicitly added to the macOS `allow-list`. Customers who participated in the beta have already been added. +While in beta, to run CI jobs on the macOS runners, GitLab SaaS customer namespaces must be explicitly added to the macOS `allow-list`. After you have been added, you can use the macOS runners for any projects in your namespace. -To request access, open a [limited availability access request](https://gitlab.com/gitlab-com/runner-saas-macos-limited-availability/-/issues/new). +To request access, open an [access request](https://gitlab.com/gitlab-com/runner-saas-macos-limited-availability/-/issues/new). The expected turnaround for activation is two business days. ## Quickstart diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index 3ab814200fb..ba3f806c96e 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -353,7 +353,7 @@ while only booting up the tested service once. ```yaml access-service: stage: build - image: docker:19.03.1 + image: docker:20.10.16 services: - docker:dind # necessary for docker run - tutum/wordpress:latest diff --git a/doc/ci/testing/accessibility_testing.md b/doc/ci/testing/accessibility_testing.md new file mode 100644 index 00000000000..7940b27acf7 --- /dev/null +++ b/doc/ci/testing/accessibility_testing.md @@ -0,0 +1,76 @@ +--- +stage: Verify +group: Pipeline Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Accessibility testing **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25144) in GitLab 12.8. + +If your application offers a web interface, you can use +[GitLab CI/CD](../index.md) to determine the accessibility +impact of pending code changes. + +[Pa11y](https://pa11y.org/) is a free and open source tool for +measuring the accessibility of web sites. GitLab integrates Pa11y into a +[CI job template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml). +The `a11y` job analyzes a defined set of web pages and reports +accessibility violations, warnings, and notices in a file named +`accessibility`. + +As of [GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73309), Pa11y uses +[WCAG 2.1 rules](https://www.w3.org/TR/WCAG21/#new-features-in-wcag-2-1). + +## Accessibility merge request widget + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39425) in GitLab 13.0 behind the disabled [feature flag](../../administration/feature_flags.md) `:accessibility_report_view`. +> - [Feature Flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/217372) in GitLab 13.1. + +GitLab displays an **Accessibility Report** in the merge request widget area: + +![Accessibility merge request widget](img/accessibility_mr_widget_v13_0.png) + +## Configure accessibility testing + +You can run Pa11y with GitLab CI/CD using the +[GitLab Accessibility Docker image](https://gitlab.com/gitlab-org/ci-cd/accessibility). + +To define the `a11y` job for GitLab 12.9 and later: + +1. [Include](../yaml/index.md#includetemplate) the + [`Accessibility.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml) + from your GitLab installation. +1. Add the following configuration to your `.gitlab-ci.yml` file. + + ```yaml + stages: + - accessibility + + variables: + a11y_urls: "https://about.gitlab.com https://gitlab.com/users/sign_in" + + include: + - template: "Verify/Accessibility.gitlab-ci.yml" + ``` + +1. Customize the `a11y_urls` variable to list the URLs of the web pages to test with Pa11y. + +The `a11y` job in your CI/CD pipeline generates these files: + +- One HTML report per URL listed in the `a11y_urls` variable. +- One file containing the collected report data. In GitLab versions 12.11 and later, this + file is named `gl-accessibility.json`. In GitLab versions 12.10 and earlier, this file + is named [`accessibility.json`](https://gitlab.com/gitlab-org/ci-cd/accessibility/-/merge_requests/9). + +You can [view job artifacts in your browser](../pipelines/job_artifacts.md#download-job-artifacts). + +NOTE: +For GitLab versions earlier than 12.9, use `include:remote` and +link to the [current template in the default branch](https://gitlab.com/gitlab-org/gitlab/-/raw/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml) + +NOTE: +The job definition provided by the template does not support Kubernetes. + +You cannot pass configurations into Pa11y via CI configuration. +To change the configuration, edit a copy of the template in your CI file. diff --git a/doc/ci/testing/browser_performance_testing.md b/doc/ci/testing/browser_performance_testing.md new file mode 100644 index 00000000000..260ecf6108d --- /dev/null +++ b/doc/ci/testing/browser_performance_testing.md @@ -0,0 +1,242 @@ +--- +stage: Verify +group: Pipeline Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Browser Performance Testing **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3507) in GitLab 10.3. + +If your application offers a web interface and you're using +[GitLab CI/CD](../index.md), you can quickly determine the rendering performance +impact of pending code changes in the browser. + +NOTE: +You can automate this feature in your applications by using [Auto DevOps](../../topics/autodevops/index.md). + +## Overview + +GitLab uses [Sitespeed.io](https://www.sitespeed.io), a free and open source +tool, for measuring the rendering performance of web sites. The +[Sitespeed plugin](https://gitlab.com/gitlab-org/gl-performance) that GitLab built outputs +the performance score for each page analyzed in a file called `browser-performance.json` +this data can be shown on Merge Requests. + +## Use cases + +Consider the following workflow: + +1. A member of the marketing team is attempting to track engagement by adding a new tool. +1. With browser performance metrics, they see how their changes are impacting the usability + of the page for end users. +1. The metrics show that after their changes, the performance score of the page has gone down. +1. When looking at the detailed report, they see the new JavaScript library was + included in `<head>`, which affects loading page speed. +1. They ask for help from a front end developer, who sets the library to load asynchronously. +1. The frontend developer approves the merge request, and authorizes its deployment to production. + +## How browser performance testing works + +First, define a job in your `.gitlab-ci.yml` file that generates the +[Browser Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsbrowser_performance). +GitLab then checks this report, compares key performance metrics for each page +between the source and target branches, and shows the information in the merge request. + +For an example Browser Performance job, see +[Configuring Browser Performance Testing](#configuring-browser-performance-testing). + +NOTE: +If the Browser Performance report has no data to compare, such as when you add the +Browser Performance job in your `.gitlab-ci.yml` for the very first time, +the Browser Performance report widget doesn't display. It must have run at least +once on the target branch (`main`, for example), before it displays in a +merge request targeting that branch. + +![Browser Performance Widget](img/browser_performance_testing.png) + +## Configuring Browser Performance Testing + +This example shows how to run the [sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) +on your code by using GitLab CI/CD and [sitespeed.io](https://www.sitespeed.io) +using Docker-in-Docker. + +1. First, set up GitLab Runner with a + [Docker-in-Docker build](../docker/using_docker_build.md#use-docker-in-docker). +1. Configure the default Browser Performance Testing CI/CD job as follows in your `.gitlab-ci.yml` file: + + ```yaml + include: + template: Verify/Browser-Performance.gitlab-ci.yml + + browser_performance: + variables: + URL: https://example.com + ``` + +WARNING: +In GitLab 13.12 and earlier, the job [was named](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) `performance`. + +The above example: + +- Creates a `browser_performance` job in your CI/CD pipeline and runs sitespeed.io against the webpage you + defined in `URL` to gather key metrics. +- Uses a template that doesn't work with Kubernetes clusters. If you are using a Kubernetes cluster, + use [`template: Jobs/Browser-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Browser-Performance-Testing.gitlab-ci.yml) + instead. +- Uses a CI/CD template that is included in all GitLab installations since 12.4. If you are using + GitLab 12.3 or earlier, you must [add the configuration manually](#gitlab-versions-132-and-earlier). + +The template uses the [GitLab plugin for sitespeed.io](https://gitlab.com/gitlab-org/gl-performance), +and it saves the full HTML sitespeed.io report as a [Browser Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsbrowser_performance) +that you can later download and analyze. This implementation always takes the latest +Browser Performance artifact available. If [GitLab Pages](../../user/project/pages/index.md) is enabled, +you can view the report directly in your browser. + +You can also customize the jobs with CI/CD variables: + +- `SITESPEED_IMAGE`: Configure the Docker image to use for the job (default `sitespeedio/sitespeed.io`), but not the image version. +- `SITESPEED_VERSION`: Configure the version of the Docker image to use for the job (default `14.1.0`). +- `SITESPEED_OPTIONS`: Configure any additional sitespeed.io options as required (default `nil`). Refer to the [sitespeed.io documentation](https://www.sitespeed.io/documentation/sitespeed.io/configuration/) for more details. + +For example, you can override the number of runs sitespeed.io +makes on the given URL, and change the version: + +```yaml +include: + template: Verify/Browser-Performance.gitlab-ci.yml + +browser_performance: + variables: + URL: https://www.sitespeed.io/ + SITESPEED_VERSION: 13.2.0 + SITESPEED_OPTIONS: -n 5 +``` + +### Configuring degradation threshold + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27599) in GitLab 13.0. + +You can configure the sensitivity of degradation alerts to avoid getting alerts for minor drops in metrics. +This is done by setting the `DEGRADATION_THRESHOLD` CI/CD variable. In the example below, the alert only shows up +if the `Total Score` metric degrades by 5 points or more: + +```yaml +include: + template: Verify/Browser-Performance.gitlab-ci.yml + +browser_performance: + variables: + URL: https://example.com + DEGRADATION_THRESHOLD: 5 +``` + +The `Total Score` metric is based on sitespeed.io's [coach performance score](https://www.sitespeed.io/documentation/sitespeed.io/metrics/#performance-score). There is more information in [the coach documentation](https://www.sitespeed.io/documentation/coach/how-to/#what-do-the-coach-do). + +### Performance testing on Review Apps + +The above CI YAML configuration is great for testing against static environments, and it can +be extended for dynamic environments, but a few extra steps are required: + +1. The `browser_performance` job should run after the dynamic environment has started. +1. In the `review` job: + 1. Generate a URL list file with the dynamic URL. + 1. Save the file as an artifact, for example with `echo $CI_ENVIRONMENT_URL > environment_url.txt` + in your job's `script`. + 1. Pass the list as the URL environment variable (which can be a URL or a file containing URLs) + to the `browser_performance` job. +1. You can now run the sitespeed.io container against the desired hostname and + paths. + +Your `.gitlab-ci.yml` file would look like: + +```yaml +stages: + - deploy + - performance + +include: + template: Verify/Browser-Performance.gitlab-ci.yml + +review: + stage: deploy + environment: + name: review/$CI_COMMIT_REF_SLUG + url: http://$CI_COMMIT_REF_SLUG.$APPS_DOMAIN + script: + - run_deploy_script + - echo $CI_ENVIRONMENT_URL > environment_url.txt + artifacts: + paths: + - environment_url.txt + only: + - branches + except: + - master + +browser_performance: + dependencies: + - review + variables: + URL: environment_url.txt +``` + +### GitLab versions 13.2 and earlier + +Browser Performance Testing has gone through several changes since its introduction. +In this section we detail these changes and how you can run the test based on your +GitLab version: + +- In 13.2 the feature was renamed from `Performance` to `Browser Performance` with additional + template CI/CD variables. +- In GitLab 12.4 [a job template was made available](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Browser-Performance.gitlab-ci.yml). +- For 11.5 to 12.3 no template is available and the job has to be defined manually as follows: + + ```yaml + performance: + stage: performance + image: docker:git + variables: + URL: https://example.com + SITESPEED_VERSION: 14.1.0 + SITESPEED_OPTIONS: '' + services: + - docker:stable-dind + script: + - mkdir gitlab-exporter + - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/1.1.0/index.js + - mkdir sitespeed-results + - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:$SITESPEED_VERSION --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL $SITESPEED_OPTIONS + - mv sitespeed-results/data/performance.json performance.json + artifacts: + paths: + - performance.json + - sitespeed-results/ + reports: + performance: performance.json + ``` + +- For 11.4 and earlier the job should be defined as follows: + + ```yaml + performance: + stage: performance + image: docker:git + variables: + URL: https://example.com + services: + - docker:stable-dind + script: + - mkdir gitlab-exporter + - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/1.1.0/index.js + - mkdir sitespeed-results + - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL + - mv sitespeed-results/data/performance.json performance.json + artifacts: + paths: + - performance.json + - sitespeed-results/ + ``` + +Upgrading to the latest version and using the templates is recommended, to ensure +you receive the latest updates, including updates to the sitespeed.io versions. diff --git a/doc/ci/testing/code_quality.md b/doc/ci/testing/code_quality.md new file mode 100644 index 00000000000..401279b9601 --- /dev/null +++ b/doc/ci/testing/code_quality.md @@ -0,0 +1,636 @@ +--- +stage: Secure +group: Static Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Code Quality **(FREE)** + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. + +To ensure your project's code stays simple, readable, and easy to contribute to, +you can use [GitLab CI/CD](../index.md) to analyze your source code quality. + +For example, while you're implementing a feature, you can run Code Quality reports +to analyze how your improvements are impacting your code's quality. You can +use this information to ensure that your changes are improving performance rather +than degrading it. + +Code Quality: + +- Uses [plugins](https://docs.codeclimate.com/docs/list-of-engines) supported by Code Climate, which are + free and open source. Code Quality does not require a Code Climate + subscription. +- Runs in [pipelines](../pipelines/index.md) by using a Docker image built in the + [GitLab Code Quality](https://gitlab.com/gitlab-org/ci-cd/codequality) project. +- Uses [default Code Climate configurations](https://gitlab.com/gitlab-org/ci-cd/codequality/-/tree/master/codeclimate_defaults). +- Can make use of a [template](#example-configuration). +- Is available by using [Auto Code Quality](../../topics/autodevops/stages.md#auto-code-quality), provided by [Auto DevOps](../../topics/autodevops/index.md). +- Can be extended through [Analysis Plugins](https://docs.codeclimate.com/docs/list-of-engines) or a [custom tool](#implementing-a-custom-tool). + +## Summary of features per tier + +Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), +as shown in the following table: + +| Capability | In Free | In Premium | In Ultimate | +|:----------------------------------------------------------------------|:--------------------|:--------------------|:-------------------| +| [Configure scanners](#configuring-jobs-using-variables) | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| [Integrate custom scanners](#implementing-a-custom-tool) | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| [Generate JSON or HTML report artifacts](#generate-an-html-report) | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| [See findings in merge request widget](#code-quality-widget) | **{check-circle}** | **{check-circle}** | **{check-circle}** | +| [See reports in CI pipelines](#code-quality-reports) | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | +| [See findings in merge request diff view](#code-quality-in-diff-view) | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | + +## Code Quality Widget + +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. + +Going a step further, GitLab can show the Code Quality report right +in the merge request widget area if a report from the target branch is available to compare to: + +![Code Quality Widget](img/code_quality_widget_13_11.png) + +Watch a quick walkthrough of Code Quality in action: + +<div class="video-fallback"> + See the video: <a href="https://www.youtube.com/watch?v=B32LxtJKo9M">Code Quality: Speed Run</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/B32LxtJKo9M" frameborder="0" allowfullscreen="true"> </iframe> +</figure> + +NOTE: +For one customer, the auditor found that having Code Quality, SAST, and Container Scanning all automated in GitLab CI/CD was almost better than a manual review! [Read more](https://about.gitlab.com/customers/bi_worldwide/). + +See also the Code Climate list of [Supported Languages for Maintainability](https://docs.codeclimate.com/docs/supported-languages-for-maintainability). + +## Code Quality in diff view **(ULTIMATE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267612) in GitLab 13.11, disabled by default behind the `codequality_mr_diff` [feature flag](../../administration/feature_flags.md). +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 13.12. +> - [Disabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) in GitLab 14.0 due to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/334116). +> - [Inline annotation added](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) and [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 14.1. + +Changes to files in merge requests can cause Code Quality to fall if merged. In these cases, +the merge request's diff view displays an indicator next to lines with new Code Quality violations. For example: + +![Code Quality MR diff report](img/code_quality_mr_diff_report_v14_2.png) + +## Example configuration + +This example shows how to run Code Quality on your code by using GitLab CI/CD and Docker. + +- Using shared runners, the job should be configured For the [Docker-in-Docker workflow](../docker/using_docker_build.md#use-docker-in-docker). +- Using private runners, there is an [alternative configuration](#set-up-a-private-runner-for-code-quality-without-docker-in-docker) recommended for running Code Quality analysis more efficiently. + +In either configuration, the runner must have enough disk space to handle generated Code Quality files. For example on the [GitLab project](https://gitlab.com/gitlab-org/gitlab) the files are approximately 7 GB. + +Once you set up GitLab Runner, include the [Code Quality template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Code-Quality.gitlab-ci.yml) in your CI configuration: + +```yaml +include: + - template: Code-Quality.gitlab-ci.yml +``` + +The above example creates a `code_quality` job in your CI/CD pipeline which +scans your source code for code quality issues. The report is saved as a +[Code Quality report artifact](../yaml/artifacts_reports.md#artifactsreportscodequality) +that you can later download and analyze. + +It's also possible to override the URL to the Code Quality image by +setting the `CODE_QUALITY_IMAGE` CI/CD variable. This is particularly useful if you want +to lock in a specific version of Code Quality, or use a fork of it: + +```yaml +include: + - template: Code-Quality.gitlab-ci.yml + +code_quality: + variables: + CODE_QUALITY_IMAGE: "registry.example.com/codequality-fork:latest" +``` + +In [GitLab 13.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/11100), you can override the [Code Quality environment variables](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables): + +```yaml +variables: + TIMEOUT_SECONDS: 1 + +include: + - template: Code-Quality.gitlab-ci.yml +``` + +By default, report artifacts are not downloadable. If you need them downloadable on the +job details page, you can add `gl-code-quality-report.json` to the artifact paths like so: + +```yaml +include: + - template: Code-Quality.gitlab-ci.yml + +code_quality: + artifacts: + paths: [gl-code-quality-report.json] +``` + +The included `code_quality` job is running in the `test` stage, so it needs to be included in your CI configuration, like so: + +```yaml +stages: + - test +``` + +NOTE: +This information is automatically extracted and shown right in the merge request widget. + +WARNING: +On self-managed instances, if a malicious actor compromises the Code Quality job +definition they could execute privileged Docker commands on the runner +host. Having proper access control policies mitigates this attack vector by +allowing access only to trusted actors. + +### Set up a private runner for code quality without Docker-in-Docker + +It's possible to configure your own runners and avoid Docker-in-Docker. You can use a +configuration that may greatly speed up job execution without requiring your runners +to operate in privileged mode. + +This alternative configuration uses socket binding to share the Runner's Docker daemon +with the job environment. Be aware that this configuration [has significant considerations](../docker/using_docker_build.md#use-docker-socket-binding) +to be consider, but may be preferable depending on your use case. + +1. Register a new runner: + + ```shell + $ gitlab-runner register --executor "docker" \ + --docker-image="docker:stable" \ + --url "https://gitlab.com/" \ + --description "cq-sans-dind" \ + --tag-list "cq-sans-dind" \ + --locked="false" \ + --access-level="not_protected" \ + --docker-volumes "/cache"\ + --docker-volumes "/builds:/builds"\ + --docker-volumes "/var/run/docker.sock:/var/run/docker.sock" \ + --registration-token="<project_token>" \ + --non-interactive + ``` + +1. **Optional, but recommended:** Set the builds directory to `/tmp/builds`, + so job artifacts are periodically purged from the runner host. If you skip + this step, you must clean up the default builds directory (`/builds`) yourself. + You can do this by adding the following two flags to `gitlab-runner register` + in the previous step. + + ```shell + --builds-dir "/tmp/builds" + --docker-volumes "/tmp/builds:/tmp/builds" # Use this instead of --docker-volumes "/builds:/builds" + ``` + + The resulting configuration: + + ```toml + [[runners]] + name = "cq-sans-dind" + url = "https://gitlab.com/" + token = "<project_token>" + executor = "docker" + builds_dir = "/tmp/builds" + [runners.docker] + tls_verify = false + image = "docker:stable" + privileged = false + disable_entrypoint_overwrite = false + oom_kill_disable = false + disable_cache = false + volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock", "/tmp/builds:/tmp/builds"] + shm_size = 0 + [runners.cache] + [runners.cache.s3] + [runners.cache.gcs] + ``` + +1. Apply two overrides to the `code_quality` job created by the template: + + ```yaml + include: + - template: Code-Quality.gitlab-ci.yml + + code_quality: + services: # Shut off Docker-in-Docker + tags: + - cq-sans-dind # Set this job to only run on our new specialized runner + ``` + +The end result is that: + +- Privileged mode is not used. +- Docker-in-Docker is not used. +- Docker images, including all CodeClimate images, are cached, and not re-fetched for subsequent jobs. + +With this configuration, the run time for a second pipeline is much shorter. For example +this [small change](https://gitlab.com/drew/test-code-quality-template/-/merge_requests/4/diffs?commit_id=1e705607aef7236c1b20bb6f637965f3f3e53a46) +to an [open merge request](https://gitlab.com/drew/test-code-quality-template/-/merge_requests/4/pipelines) +running Code Quality analysis ran significantly faster the second time: + +![Code Quality sequential runs without DinD](img/code_quality_host_bound_sequential.png) + +This configuration is not possible on `gitlab.com` shared runners. Shared runners +are configured with `privileged=true`, and they do not expose `docker.sock` into +the job container. As a result, socket binding cannot be used to make `docker` available +in the context of the job script. + +[Docker-in-Docker](../docker/using_docker_build.md#use-docker-in-docker) +was chosen as an operational decision by the runner team, instead of exposing `docker.sock`. + +### Disabling the code quality job + +The `code_quality` job doesn't run if the `$CODE_QUALITY_DISABLED` CI/CD variable +is present. Please refer to the CI/CD variables [documentation](../variables/index.md) +to learn more about how to define one. + +To disable the `code_quality` job, add `CODE_QUALITY_DISABLED` as a custom CI/CD variable. +This can be done: + +- For [the whole project](../variables/index.md#custom-cicd-variables). +- For a single pipeline run: + + 1. Go to **CI/CD > Pipelines** + 1. Select **Run pipeline** + 1. Add `CODE_QUALITY_DISABLED` as the variable key, with any value. + +### Using with merge request pipelines + +The configuration provided by the Code Quality template does not let the `code_quality` job +run on [merge request pipelines](../pipelines/merge_request_pipelines.md). + +If merge request pipelines is enabled, the `code_quality:rules` must be redefined. + +The template has these [`rules`](../yaml/index.md#rules) for the `code quality` job: + +```yaml +code_quality: + rules: + - if: $CODE_QUALITY_DISABLED + when: never + - if: $CI_COMMIT_TAG || $CI_COMMIT_BRANCH +``` + +If you are using merge request pipelines, your `rules` (or [`workflow: rules`](../yaml/index.md#workflow)) +might look like this example: + +```yaml +job1: + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" # Run job1 in merge request pipelines + - if: $CI_COMMIT_BRANCH == "main" # Run job1 in pipelines on the main branch (but not in other branch pipelines) + - if: $CI_COMMIT_TAG # Run job1 in pipelines for tags +``` + +To make these work together, you need to overwrite the code quality `rules` +so that they match your current `rules`. From the example above, it could look like: + +```yaml +include: + - template: Code-Quality.gitlab-ci.yml + +code_quality: + rules: + - if: $CODE_QUALITY_DISABLED + when: never + - if: $CI_PIPELINE_SOURCE == "merge_request_event" # Run code quality job in merge request pipelines + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run code quality job in pipelines on the default branch (but not in other branch pipelines) + - if: $CI_COMMIT_TAG # Run code quality job in pipelines for tags +``` + +### Configure Code Quality to use a private container image registry + +> [Introduced](https://gitlab.com/gitlab-org/ci-cd/codequality/-/merge_requests/30) in 13.7. + +To reduce network time and external dependency, you can use your own +container image registry to host the Code Quality Docker images. Because of +the nested architecture of container execution, the registry prefix must +be specifically configured to be passed down into CodeClimate's subsequent +`docker pull` commands for individual engines. + +The following two variables can address all of the required image pulls: + +- `CODE_QUALITY_IMAGE`: A fully prefixed image name that can be located anywhere + accessible from your job environment. GitLab Container Registry can be used here + to host your own copy. +- `CODECLIMATE_PREFIX`: The domain of your intended container image registry. This + is a configuration option supported by [CodeClimate CLI](https://github.com/codeclimate/codeclimate/pull/948). You must: + - Include a trailing slash (`/`). + - Not include a protocol prefix, such as `https://`. + +```yaml +include: + - template: Jobs/Code-Quality.gitlab-ci.yml + +code_quality: + variables: + CODE_QUALITY_IMAGE: "my-private-registry.local:12345/codequality:0.85.24" + CODECLIMATE_PREFIX: "my-private-registry.local:12345/" +``` + +The images in the private container image registry must be available without authentication. +Follow [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/355814) for more information. + +This example is specific to GitLab Code Quality. For more general +instructions on how to configure DinD with a registry mirror, see the +relevant [documentation](../docker/using_docker_build.md#enable-registry-mirror-for-dockerdind-service). + +## Configuring jobs using variables + +The Code Quality job supports environment variables that users can set to +configure job execution at runtime. + +For a list of available environment variables, see +[Environment variables](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables). + +## Implementing a custom tool + +It's possible to have a custom tool provide Code Quality reports in GitLab. To +do this: + +1. Define a job in your `.gitlab-ci.yml` file that generates the + [Code Quality report artifact](../yaml/artifacts_reports.md#artifactsreportscodequality). +1. Configure your tool to generate the Code Quality report artifact as a JSON + file that implements a subset of the [Code Climate spec](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types). + +The Code Quality report artifact JSON file must contain an array of objects +with the following properties: + +| Name | Description | +| ---------------------- | ----------------------------------------------------------------------------------------- | +| `description` | A description of the code quality violation. | +| `fingerprint` | A unique fingerprint to identify the code quality violation. For example, an MD5 hash. | +| `severity` | A severity string (can be `info`, `minor`, `major`, `critical`, or `blocker`). | +| `location.path` | The relative path to the file containing the code quality violation. | +| `location.lines.begin` or `location.positions.begin.line` | The line on which the code quality violation occurred. | + +Example: + +```json +[ + { + "description": "'unused' is assigned a value but never used.", + "fingerprint": "7815696ecbf1c96e6894b779456d330e", + "severity": "minor", + "location": { + "path": "lib/index.js", + "lines": { + "begin": 42 + } + } + } +] +``` + +NOTE: +Although the Code Climate spec supports more properties, those are ignored by +GitLab. +The GitLab parser does not allow a [byte order mark](https://en.wikipedia.org/wiki/Byte_order_mark) +at the beginning of the file. + +## Code Quality reports **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21527) in GitLab 12.9. + +![Code Quality Report](img/code_quality_report_13_11.png) + +After the Code Quality job completes: + +- Potential changes to code quality are shown directly in the merge request. + The Code Quality widget in the merge request compares the reports from the base and head of the branch, + then lists any violations that are resolved or created when the branch is merged. +- The full JSON report is available as a + [downloadable artifact](../pipelines/job_artifacts.md#download-job-artifacts) + for the `code_quality` job. +- The full list of code quality violations generated by a pipeline is shown in the + Code Quality tab of the Pipeline Details page. + +## Generate an HTML report + +In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/ci-cd/codequality/-/issues/10), +it is possible to generate an HTML report file by setting the `REPORT_FORMAT` +CI/CD variable to `html`. This is useful if you just want to view the report in a more +human-readable format or to publish this artifact on GitLab Pages for even +easier reviewing. + +To generate both JSON and HTML report files, add another job to your template by using `extends: code_quality`: + +```yaml +include: + - template: Code-Quality.gitlab-ci.yml + +code_quality_html: + extends: code_quality + variables: + REPORT_FORMAT: html + artifacts: + paths: [gl-code-quality-report.html] +``` + +NOTE: +Adding a job means your code is scanned twice: once to generate a JSON report and once to generate an HTML report. + +You can also generate _only_ an HTML report instead of the standard JSON report. To do so, set `REPORT_FORMAT` to `html` in the existing job: + +```yaml +include: + - template: Code-Quality.gitlab-ci.yml + +code_quality: + variables: + REPORT_FORMAT: html + artifacts: + paths: [gl-code-quality-report.html] +``` + +WARNING: +If you only generate an HTML report, you can't see your results in the [merge request widget](#code-quality-widget), [pipeline report](#code-quality-reports), or [diff view](#code-quality-in-diff-view). +These features require a JSON report. + +## Extending functionality + +### Using Analysis Plugins + +Should there be a need to extend the default functionality provided by Code Quality, as stated in [Code Quality](#code-quality), [Analysis Plugins](https://docs.codeclimate.com/docs/list-of-engines) are available. + +For example, to use the [SonarJava analyzer](https://docs.codeclimate.com/docs/sonar-java), +add a file named `.codeclimate.yml` containing the [enablement code](https://docs.codeclimate.com/docs/sonar-java#enable-the-plugin) +for the plugin to the root of your repository: + +```yaml +version: "2" +plugins: + sonar-java: + enabled: true +``` + +This adds SonarJava to the `plugins:` section of the [default `.codeclimate.yml`](https://gitlab.com/gitlab-org/ci-cd/codequality/-/blob/master/codeclimate_defaults/.codeclimate.yml.template) +included in your project. + +Changes to the `plugins:` section do not affect the `exclude_patterns` section of the +default `.codeclimate.yml`. See the Code Climate documentation for +[excluding files and folders](https://docs.codeclimate.com/docs/excluding-files-and-folders) +for more details. + +Here's [an example project](https://gitlab.com/jheimbuck_gl/jh_java_example_project) that uses Code Quality with a `.codeclimate.yml` file. + +## Use a Code Quality image hosted in a registry with untrusted certificates + +If you set the `CODE_QUALITY_IMAGE` to an image that is hosted in a +Docker registry which uses a TLS certificate that is not trusted, such as +a self-signed certificate, you can see errors like the one below: + +```shell +$ docker pull --quiet "$CODE_QUALITY_IMAGE" +Error response from daemon: Get https://gitlab.example.com/v2/: x509: certificate signed by unknown authority +``` + +To fix this, configure the Docker daemon to [trust certificates](https://docs.docker.com/registry/insecure/#use-self-signed-certificates) +by putting the certificate inside of the `/etc/docker/certs.d` +directory. + +This Docker daemon is exposed to the subsequent Code Quality Docker container in the +[GitLab Code Quality template](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.8.3-ee/lib/gitlab/ci/templates/Jobs/Code-Quality.gitlab-ci.yml#L41) +and should be to exposed any other containers in which you want to have +your certificate configuration apply. + +### Docker + +If you have access to GitLab Runner configuration, add the directory as a +[volume mount](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#volumes-in-the-runnersdocker-section). For example: + +```toml +[[runners]] + ... + executor = "docker" + [runners.docker] + ... + privileged = true + volumes = ["/cache", "/etc/gitlab-runner/certs/gitlab.example.com.crt:/etc/docker/certs.d/gitlab.example.com/ca.crt:ro"] +``` + +Replace `gitlab.example.com` with the actual domain of the registry. + +### Kubernetes + +If you have access to GitLab Runner configuration and the Kubernetes cluster, +you can [mount a ConfigMap](https://docs.gitlab.com/runner/executors/kubernetes.html#configmap-volumes): + +1. Create a ConfigMap with the certificate: + + ```shell + kubectl create configmap registry-crt --namespace gitlab-runner --from-file /etc/gitlab-runner/certs/gitlab.example.com.crt + ``` + +1. Update GitLab Runner `config.toml` to specify the ConfigMap: + + ```toml + [[runners]] + ... + executor = "kubernetes" + [runners.kubernetes] + image = "alpine:3.12" + privileged = true + [[runners.kubernetes.volumes.config_map]] + name = "registry-crt" + mount_path = "/etc/docker/certs.d/gitlab.example.com/ca.crt" + sub_path = "gitlab.example.com.crt" + ``` + +Replace `gitlab.example.com` with the actual domain of the registry. + +## Troubleshooting + +### Changing the default configuration has no effect + +A common issue is that the terms `Code Quality` (GitLab specific) and `Code Climate` +(Engine used by GitLab) are very similar. You must add a **`.codeclimate.yml`** file +to change the default configuration, **not** a `.codequality.yml` file. If you use +the wrong filename, the [default `.codeclimate.yml`](https://gitlab.com/gitlab-org/ci-cd/codequality/-/blob/master/codeclimate_defaults/.codeclimate.yml.template) +is still used. + +### No Code Quality report is displayed in a merge request + +This can be due to multiple reasons: + +- You just added the Code Quality job in your `.gitlab-ci.yml`. The report does not + have anything to compare to yet, so no information can be displayed. It only displays + after future merge requests have something to compare to. +- Your pipeline is not set to run the code quality job on your target branch. If there is no report generated from the target branch, your MR branch reports have nothing to compare to. In this situation you will see an error stating `Base pipeline codequality artifact not found`. +- If no [degradation or error is detected](https://docs.codeclimate.com/docs/maintainability#section-checks), + nothing is displayed. +- The [`artifacts:expire_in`](../yaml/index.md#artifactsexpire_in) CI/CD + setting can cause the Code Quality artifacts to expire faster than desired. +- The widgets use the pipeline of the latest commit to the target branch. If commits are made to the default branch that do not run the code quality job, this may cause the merge request widget to have no base report for comparison. +- If you use the [`REPORT_STDOUT` environment variable](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables), no report file is generated and nothing displays in the merge request. +- Large `gl-code-quality-report.json` files (esp. >10 MB) are [known to prevent the report from being displayed](https://gitlab.com/gitlab-org/gitlab/-/issues/2737). + As a work-around, try removing [properties](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types) + that are [ignored by GitLab](#implementing-a-custom-tool). You can: + - Configure the Code Quality tool to not output those types. + - Use `sed`, `awk` or similar commands in the `.gitlab-ci.yml` script to + edit the `gl-code-quality-report.json` before the job completes. + +### Only a single Code Quality report is displayed, but more are defined + +GitLab only uses the Code Quality artifact from the latest created job (with the largest job ID). +If multiple jobs in a pipeline generate a code quality artifact, those of earlier jobs are ignored. +To avoid confusion, configure only one job to generate a `gl-code-quality-report.json`. + +### RuboCop errors + +When using Code Quality jobs on a Ruby project, you can encounter problems running RuboCop. +For example, the following error can appear when using either a very recent or very old version +of Ruby: + +```plaintext +/usr/local/bundle/gems/rubocop-0.52.1/lib/rubocop/config.rb:510:in `check_target_ruby': +Unknown Ruby version 2.7 found in `.ruby-version`. (RuboCop::ValidationError) +Supported versions: 2.1, 2.2, 2.3, 2.4, 2.5 +``` + +This is caused by the default version of RuboCop used by the check engine not covering +support for the Ruby version in use. + +To use a custom version of RuboCop that +[supports the version of Ruby used by the project](https://docs.rubocop.org/rubocop/compatibility.html#support-matrix), +you can [override the configuration through a `.codeclimate.yml` file](https://docs.codeclimate.com/docs/rubocop#using-rubocops-newer-versions) +created in the project repository. + +For example, to specify using RuboCop release **0.67**: + +```yaml +version: "2" +plugins: + rubocop: + enabled: true + channel: rubocop-0-67 +``` + +### No Code Quality appears on merge requests when using custom tool + +If your merge requests do not show any code quality changes when using a custom tool, +ensure that the line property is an `integer`. + +### Code Quality CI job with Code Climate plugins enabled fails with error + +If you enabled any of the Code Climate plugins, and the Code Quality CI job fails with the error +below, it's likely the job takes longer than the default timeout of 900 seconds: + +```shell +error: (CC::CLI::Analyze::EngineFailure) engine pmd ran for 900 seconds and was killed +Could not analyze code quality for the repository at /code +``` + +To work around this problem, set `TIMEOUT_SECONDS` to a higher value in your `.gitlab.-ci.yml` file. + +For example: + +```yaml +variables: + TIMEOUT_SECONDS: 3600 +``` diff --git a/doc/ci/testing/fail_fast_testing.md b/doc/ci/testing/fail_fast_testing.md new file mode 100644 index 00000000000..7b95b1ac54a --- /dev/null +++ b/doc/ci/testing/fail_fast_testing.md @@ -0,0 +1,97 @@ +--- +stage: Verify +group: Pipeline Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Fail Fast Testing **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198550) in GitLab 13.1. + +For applications that use RSpec for running tests, we've introduced the `Verify/Failfast` +[template to run subsets of your test suite](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Verify/FailFast.gitlab-ci.yml), +based on the changes in your merge request. + +The template uses the [test_file_finder (`tff`) gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder/) +that accepts a list of files as input, and returns a list of spec (test) files +that it believes to be relevant to the input files. + +`tff` is designed for Ruby on Rails projects, so the `Verify/FailFast` template is +configured to run when changes to Ruby files are detected. By default, it runs in +the [`.pre` stage](../yaml/index.md#stage-pre) of a GitLab CI/CD pipeline, +before all other stages. + +## Example use case + +Fail fast testing is useful when adding new functionality to a project and adding +new automated tests. + +Your project could have hundreds of thousands of tests that take a long time to complete. +You may be confident that a new test will pass, but you have to wait for all the tests +to complete to verify it. This could take an hour or more, even when using parallelization. + +Fail fast testing gives you a faster feedback loop from the pipeline. It lets you +know quickly that the new tests are passing and the new functionality did not break +other tests. + +## Requirements + +This template requires: + +- A project built in Rails that uses RSpec for testing. +- CI/CD configured to: + - Use a Docker image with Ruby available. + - Use [Merge request pipelines](../pipelines/merge_request_pipelines.md#prerequisites) +- [Merged results pipelines](../pipelines/merged_results_pipelines.md#enable-merged-results-pipelines) + enabled in the project settings. +- A Docker image with Ruby available. The template uses `image: ruby:2.6` by default, but you [can override](../yaml/includes.md#override-included-configuration-values) this. + +## Configuring Fast RSpec Failure + +We use the following plain RSpec configuration as a starting point. It installs all the +project gems and executes `rspec`, on merge request pipelines only. + +```yaml +rspec-complete: + stage: test + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + script: + - bundle install + - bundle exec rspec +``` + +To run the most relevant specs first instead of the whole suite, [`include`](../yaml/index.md#include) +the template by adding the following to your CI/CD configuration: + +```yaml +include: + - template: Verify/FailFast.gitlab-ci.yml +``` + +To customize the job, specific options may be set to override the template. For example, to override the default Docker image: + +```yaml +include: + - template: Verify/FailFast.gitlab-ci.yml + +rspec-rails-modified-path-specs: + image: custom-docker-image-with-ruby +``` + +### Example test loads + +For illustrative purposes, let's say our Rails app spec suite consists of 100 specs per model for ten models. + +If no Ruby files are changed: + +- `rspec-rails-modified-paths-specs` does not run any tests. +- `rspec-complete` runs the full suite of 1000 tests. + +If one Ruby model is changed, for example `app/models/example.rb`, then `rspec-rails-modified-paths-specs` +runs the 100 tests for `example.rb`: + +- If all of these 100 tests pass, then the full `rspec-complete` suite of 1000 tests is allowed to run. +- If any of these 100 tests fail, they fail quickly, and `rspec-complete` does not run any tests. + +The final case saves resources and time as the full 1000 test suite does not run. diff --git a/doc/user/project/merge_requests/img/accessibility_mr_widget_v13_0.png b/doc/ci/testing/img/accessibility_mr_widget_v13_0.png Binary files differindex 4ada7e25b65..4ada7e25b65 100644 --- a/doc/user/project/merge_requests/img/accessibility_mr_widget_v13_0.png +++ b/doc/ci/testing/img/accessibility_mr_widget_v13_0.png diff --git a/doc/user/project/merge_requests/img/browser_performance_testing.png b/doc/ci/testing/img/browser_performance_testing.png Binary files differindex a3d7022bcfc..a3d7022bcfc 100644 --- a/doc/user/project/merge_requests/img/browser_performance_testing.png +++ b/doc/ci/testing/img/browser_performance_testing.png diff --git a/doc/user/project/merge_requests/img/code_quality_host_bound_sequential.png b/doc/ci/testing/img/code_quality_host_bound_sequential.png Binary files differindex 2b31f3b42ee..2b31f3b42ee 100644 --- a/doc/user/project/merge_requests/img/code_quality_host_bound_sequential.png +++ b/doc/ci/testing/img/code_quality_host_bound_sequential.png diff --git a/doc/user/project/merge_requests/img/code_quality_mr_diff_report_v14_2.png b/doc/ci/testing/img/code_quality_mr_diff_report_v14_2.png Binary files differindex c1e9aad24ac..c1e9aad24ac 100644 --- a/doc/user/project/merge_requests/img/code_quality_mr_diff_report_v14_2.png +++ b/doc/ci/testing/img/code_quality_mr_diff_report_v14_2.png diff --git a/doc/user/project/merge_requests/img/code_quality_report_13_11.png b/doc/ci/testing/img/code_quality_report_13_11.png Binary files differindex 36341548328..36341548328 100644 --- a/doc/user/project/merge_requests/img/code_quality_report_13_11.png +++ b/doc/ci/testing/img/code_quality_report_13_11.png diff --git a/doc/user/project/merge_requests/img/code_quality_widget_13_11.png b/doc/ci/testing/img/code_quality_widget_13_11.png Binary files differindex 57978a0ed96..57978a0ed96 100644 --- a/doc/user/project/merge_requests/img/code_quality_widget_13_11.png +++ b/doc/ci/testing/img/code_quality_widget_13_11.png diff --git a/doc/user/project/merge_requests/img/load_performance_testing.png b/doc/ci/testing/img/load_performance_testing.png Binary files differindex d5623867ee7..d5623867ee7 100644 --- a/doc/user/project/merge_requests/img/load_performance_testing.png +++ b/doc/ci/testing/img/load_performance_testing.png diff --git a/doc/ci/img/metrics_reports_advanced_v13_0.png b/doc/ci/testing/img/metrics_reports_advanced_v13_0.png Binary files differindex e96fdcf620a..e96fdcf620a 100644 --- a/doc/ci/img/metrics_reports_advanced_v13_0.png +++ b/doc/ci/testing/img/metrics_reports_advanced_v13_0.png diff --git a/doc/ci/img/metrics_reports_v13_0.png b/doc/ci/testing/img/metrics_reports_v13_0.png Binary files differindex 1597031db0b..1597031db0b 100644 --- a/doc/ci/img/metrics_reports_v13_0.png +++ b/doc/ci/testing/img/metrics_reports_v13_0.png diff --git a/doc/user/project/merge_requests/img/test_coverage_visualization_v12_9.png b/doc/ci/testing/img/test_coverage_visualization_v12_9.png Binary files differindex 1922a566dd5..1922a566dd5 100644 --- a/doc/user/project/merge_requests/img/test_coverage_visualization_v12_9.png +++ b/doc/ci/testing/img/test_coverage_visualization_v12_9.png diff --git a/doc/ci/testing/img/unit_test_report_screenshot_v13_12.png b/doc/ci/testing/img/unit_test_report_screenshot_v13_12.png Binary files differnew file mode 100644 index 00000000000..d5b1775b4ca --- /dev/null +++ b/doc/ci/testing/img/unit_test_report_screenshot_v13_12.png diff --git a/doc/ci/testing/index.md b/doc/ci/testing/index.md index 52af329873f..a8f06ec695c 100644 --- a/doc/ci/testing/index.md +++ b/doc/ci/testing/index.md @@ -9,18 +9,18 @@ info: To determine the technical writer assigned to the Stage/Group associated w Use GitLab CI/CD to test the changes included in a feature branch. You can also display reports or link to important information directly from [merge requests](../../user/project/merge_requests/index.md). -| Feature | Description | -|-------------------------------------------------------------------------------------------------|-------------| -| [Accessibility Testing](../../user/project/merge_requests/accessibility_testing.md) | Automatically report A11y violations for changed pages in merge requests. | -| [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | -| [Load Performance Testing](../../user/project/merge_requests/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | -| [Code Quality](../../user/project/merge_requests/code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. | -| [Display arbitrary job artifacts](../yaml/index.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../pipelines/job_artifacts.md) in merge requests. | -| [Unit test reports](unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire job log. | -| [License Compliance](../../user/compliance/license_compliance/index.md) | Manage the licenses of your dependencies. | -| [Metrics Reports](../metrics_reports.md) | Display the Metrics Report on the merge request so that it's fast and easier to identify changes to important metrics. | -| [Test Coverage visualization](../../user/project/merge_requests/test_coverage_visualization.md) | See test coverage results for merge requests, in the file diff. | -| [Fail fast testing](../../user/project/merge_requests/fail_fast_testing.md#fail-fast-testing) | Run a subset of your RSpec test suite, so failed tests stop the pipeline before the full suite of tests run, saving resources. | +| Feature | Description | +|-------------------------------------------------------------------------|-------------| +| [Accessibility Testing](accessibility_testing.md) | Automatically report A11y violations for changed pages in merge requests. | +| [Browser Performance Testing](browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | +| [Load Performance Testing](load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | +| [Code Quality](code_quality.md) | Analyze your source code quality using the [Code Climate](https://codeclimate.com/) analyzer and show the Code Climate report right in the merge request widget area. | +| [Display arbitrary job artifacts](../yaml/index.md#artifactsexpose_as) | Configure CI pipelines with the `artifacts:expose_as` parameter to directly link to selected [artifacts](../pipelines/job_artifacts.md) in merge requests. | +| [Unit test reports](unit_test_reports.md) | Configure your CI jobs to use Unit test reports, and let GitLab display a report on the merge request so that it's easier and faster to identify the failure without having to check the entire job log. | +| [License Compliance](../../user/compliance/license_compliance/index.md) | Manage the licenses of your dependencies. | +| [Metrics Reports](metrics_reports.md) | Display the Metrics Report on the merge request so that it's fast and easier to identify changes to important metrics. | +| [Test Coverage visualization](test_coverage_visualization.md) | See test coverage results for merge requests, in the file diff. | +| [Fail fast testing](fail_fast_testing.md) | Run a subset of your RSpec test suite, so failed tests stop the pipeline before the full suite of tests run, saving resources. | ## Security Reports **(ULTIMATE)** diff --git a/doc/ci/testing/load_performance_testing.md b/doc/ci/testing/load_performance_testing.md new file mode 100644 index 00000000000..ae177958beb --- /dev/null +++ b/doc/ci/testing/load_performance_testing.md @@ -0,0 +1,201 @@ +--- +stage: Verify +group: Pipeline Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Load Performance Testing **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10683) in GitLab 13.2. + +With Load Performance Testing, you can test the impact of any pending code changes +to your application's backend in [GitLab CI/CD](../index.md). + +GitLab uses [k6](https://k6.io/), a free and open source +tool, for measuring the system performance of applications under +load. + +Unlike [Browser Performance Testing](browser_performance_testing.md), which is +used to measure how web sites perform in client browsers, Load Performance Testing +can be used to perform various types of [load tests](https://k6.io/docs/#use-cases) +against application endpoints such as APIs, Web Controllers, and so on. +This can be used to test how the backend or the server performs at scale. + +For example, you can use Load Performance Testing to perform many concurrent +GET calls to a popular API endpoint in your application to see how it performs. + +## How Load Performance Testing works + +First, define a job in your `.gitlab-ci.yml` file that generates the +[Load Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsload_performance). +GitLab checks this report, compares key load performance metrics +between the source and target branches, and then shows the information in a merge request widget: + +![Load Performance Widget](img/load_performance_testing.png) + +Next, you need to configure the test environment and write the k6 test. + +The key performance metrics that the merge request widget shows after the test completes are: + +- Checks: The percentage pass rate of the [checks](https://k6.io/docs/using-k6/checks) configured in the k6 test. +- TTFB P90: The 90th percentile of how long it took to start receiving responses, aka the [Time to First Byte](https://en.wikipedia.org/wiki/Time_to_first_byte) (TTFB). +- TTFB P95: The 95th percentile for TTFB. +- RPS: The average requests per second (RPS) rate the test was able to achieve. + +NOTE: +If the Load Performance report has no data to compare, such as when you add the +Load Performance job in your `.gitlab-ci.yml` for the very first time, +the Load Performance report widget doesn't display. It must have run at least +once on the target branch (`main`, for example), before it displays in a +merge request targeting that branch. + +## Configure the Load Performance Testing job + +Configuring your Load Performance Testing job can be broken down into several distinct parts: + +- Determine the test parameters such as throughput, and so on. +- Set up the target test environment for load performance testing. +- Design and write the k6 test. + +### Determine the test parameters + +The first thing you need to do is determine the [type of load test](https://k6.io/docs/test-types/introduction) +you want to run, and how it will run (for example, the number of users, throughput, and so on). + +Refer to the [k6 docs](https://k6.io/docs/), especially the [k6 testing guides](https://k6.io/docs/testing-guides), +for guidance on the above and more. + +### Test Environment setup + +A large part of the effort around load performance testing is to prepare the target test environment +for high loads. You should ensure it's able to handle the +[throughput](https://k6.io/blog/monthly-visits-concurrent-users) it will be tested with. + +It's also typically required to have representative test data in the target environment +for the load performance test to use. + +We strongly recommend [not running these tests against a production environment](https://k6.io/our-beliefs#load-test-in-a-pre-production-environment). + +### Write the load performance test + +After the environment is prepared, you can write the k6 test itself. k6 is a flexible +tool and can be used to run [many kinds of performance tests](https://k6.io/docs/test-types/introduction). +Refer to the [k6 documentation](https://k6.io/docs/) for detailed information on how to write tests. + +### Configure the test in GitLab CI/CD + +When your k6 test is ready, the next step is to configure the load performance +testing job in GitLab CI/CD. The easiest way to do this is to use the +[`Verify/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Load-Performance-Testing.gitlab-ci.yml) +template that is included with GitLab. + +NOTE: +For large scale k6 tests you need to ensure the GitLab Runner instance performing the actual +test is able to handle running the test. Refer to [k6's guidance](https://k6.io/docs/testing-guides/running-large-tests#hardware-considerations) +for spec details. The [default shared GitLab.com runners](../runners/saas/linux_saas_runner.md) +likely have insufficient specs to handle most large k6 tests. + +This template runs the +[k6 Docker container](https://hub.docker.com/r/loadimpact/k6/) in the job and provides several ways to customize the +job. + +An example configuration workflow: + +1. Set up GitLab Runner to run Docker containers, like the + [Docker-in-Docker workflow](../docker/using_docker_build.md#use-docker-in-docker). +1. Configure the default Load Performance Testing CI/CD job in your `.gitlab-ci.yml` file. + You need to include the template and configure it with CI/CD variables: + + ```yaml + include: + template: Verify/Load-Performance-Testing.gitlab-ci.yml + + load_performance: + variables: + K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT> + ``` + +The above example creates a `load_performance` job in your CI/CD pipeline that runs +the k6 test. + +NOTE: +For Kubernetes setups a different template should be used: [`Jobs/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Load-Performance-Testing.gitlab-ci.yml). + +k6 has [various options](https://k6.io/docs/using-k6/options) to configure how it will run tests, such as what throughput (RPS) to run with, +how long the test should run, and so on. Almost all options can be configured in the test itself, but as +you can also pass command line options via the `K6_OPTIONS` variable. + +For example, you can override the duration of the test with a CLI option: + +```yaml + include: + template: Verify/Load-Performance-Testing.gitlab-ci.yml + + load_performance: + variables: + K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT> + K6_OPTIONS: '--duration 30s' +``` + +GitLab only displays the key performance metrics in the MR widget if k6's results are saved +via [summary export](https://k6.io/docs/results-visualization/json#summary-export) +as a [Load Performance report artifact](../yaml/artifacts_reports.md#artifactsreportsload_performance). +The latest Load Performance artifact available is always used, using the +summary values from the test. + +If [GitLab Pages](../../user/project/pages/index.md) is enabled, you can view the report directly in your browser. + +### Load Performance testing in Review Apps + +The CI/CD YAML configuration example above works for testing against static environments, +but it can be extended to work with [review apps](../review_apps/index.md) or +[dynamic environments](../environments/index.md) with a few extra steps. + +The best approach is to capture the dynamic URL in a [`.env` file](https://docs.docker.com/compose/env-file/) +as a job artifact to be shared, then use a custom CI/CD variable we've provided named `K6_DOCKER_OPTIONS` +to configure the k6 Docker container to use the file. With this, k6 can then use any +environment variables from the `.env` file in scripts using standard JavaScript, +such as: ``http.get(`${__ENV.ENVIRONMENT_URL}`)``. + +For example: + +1. In the `review` job: + 1. Capture the dynamic URL and save it into a `.env` file, for example, `echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env`. + 1. Set the `.env` file to be a [job artifact](../pipelines/job_artifacts.md). +1. In the `load_performance` job: + 1. Set it to depend on the review job, so it inherits the environment file. + 1. Set the `K6_DOCKER_OPTIONS` variable with the [Docker CLI option for environment files](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file), for example `--env-file review.env`. +1. Configure the k6 test script to use the environment variable in it's steps. + +Your `.gitlab-ci.yml` file might be similar to: + +```yaml +stages: + - deploy + - performance + +include: + template: Verify/Load-Performance-Testing.gitlab-ci.yml + +review: + stage: deploy + environment: + name: review/$CI_COMMIT_REF_SLUG + url: http://$CI_ENVIRONMENT_SLUG.example.com + script: + - run_deploy_script + - echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env + artifacts: + paths: + - review.env + rules: + - if: $CI_COMMIT_BRANCH # Modify to match your pipeline rules, or use `only/except` if needed. + +load_performance: + dependencies: + - review + variables: + K6_DOCKER_OPTIONS: '--env-file review.env' + rules: + - if: $CI_COMMIT_BRANCH # Modify to match your pipeline rules, or use `only/except` if needed. +``` diff --git a/doc/ci/testing/metrics_reports.md b/doc/ci/testing/metrics_reports.md new file mode 100644 index 00000000000..e855074ddea --- /dev/null +++ b/doc/ci/testing/metrics_reports.md @@ -0,0 +1,68 @@ +--- +stage: Verify +group: Pipeline Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Metrics Reports **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9788) in GitLab 11.10. Requires GitLab Runner 11.10 and above. + +GitLab provides a lot of great reporting tools for things like [merge requests](../../user/project/merge_requests/index.md) - [Unit test reports](unit_test_reports.md), [code quality](code_quality.md), and performance tests. While JUnit is a great open framework for tests that "pass" or "fail", it is also important to see other types of metrics from a given change. + +You can configure your job to use custom Metrics Reports, and GitLab displays a report on the merge request so that it's easier and faster to identify changes without having to check the entire log. + +![Metrics Reports](img/metrics_reports_v13_0.png) + +## Use cases + +Consider the following examples of data that can use Metrics Reports: + +1. Memory usage +1. Load testing results +1. Code complexity +1. Code coverage stats + +## How it works + +Metrics for a branch are read from the latest metrics report artifact (default filename: `metrics.txt`) as string values. + +For an MR, the values of these metrics from the feature branch are compared to the values from the target branch. Then they are displayed in the MR widget in this order: + +- Existing metrics with changed values. +- Metrics that have been added by the MR. Marked with a **New** badge. +- Metrics that have been removed by the MR. Marked with a **Removed** badge. +- Existing metrics with unchanged values. + +## How to set it up + +Add a job that creates a [metrics report](../yaml/artifacts_reports.md#artifactsreportsmetrics) (default filename: `metrics.txt`). The file should conform to the [OpenMetrics](https://openmetrics.io/) format. + +For example: + +```yaml +metrics: + script: + - echo 'metric_name metric_value' > metrics.txt + artifacts: + reports: + metrics: metrics.txt +``` + +## Advanced Example + +An advanced example of an OpenMetrics text file (from the [Prometheus documentation](https://github.com/prometheus/docs/blob/master/content/docs/instrumenting/exposition_formats.md#text-format-example)) +renders in the merge request widget as: + +![Metrics Reports Advanced](img/metrics_reports_advanced_v13_0.png) + +## Troubleshooting + +### Metrics reports did not change + +You can see `Metrics reports did not change` when trying to view metrics reports in merge requests. Reasons for this are: + +- The target branch for the merge request doesn't have a baseline metrics report for comparison. +- You don't have GitLab license at the Premium tier or above. + +There is [an issue open](https://gitlab.com/gitlab-org/gitlab/-/issues/343065) to improve this message. diff --git a/doc/ci/testing/test_coverage_visualization.md b/doc/ci/testing/test_coverage_visualization.md new file mode 100644 index 00000000000..472cfca99be --- /dev/null +++ b/doc/ci/testing/test_coverage_visualization.md @@ -0,0 +1,435 @@ +--- +stage: Verify +group: Pipeline Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Test coverage visualization **(FREE)** + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249811) in GitLab 13.5. + +With the help of [GitLab CI/CD](../index.md), you can collect the test +coverage information of your favorite testing or coverage-analysis tool, and visualize +this information inside the file diff view of your merge requests (MRs). This will allow you +to see which lines are covered by tests, and which lines still require coverage, before the +MR is merged. + +![Test Coverage Visualization Diff View](img/test_coverage_visualization_v12_9.png) + +## How test coverage visualization works + +Collecting the coverage information is done via GitLab CI/CD's +[artifacts reports feature](../yaml/index.md#artifactsreports). +You can specify one or more coverage reports to collect, including wildcard paths. +GitLab then takes the coverage information in all the files and combines it +together. Coverage files are parsed in a background job so there can be a delay +between pipeline completion and the visualization loading on the page. + +For the coverage analysis to work, you have to provide a properly formatted +[Cobertura XML](https://cobertura.github.io/cobertura/) report to +[`artifacts:reports:coverage_report`](../yaml/artifacts_reports.md#artifactsreportscoverage_report). +This format was originally developed for Java, but most coverage analysis frameworks +for other languages have plugins to add support for it, like: + +- [simplecov-cobertura](https://rubygems.org/gems/simplecov-cobertura) (Ruby) +- [gocover-cobertura](https://github.com/boumenot/gocover-cobertura) (Golang) + +Other coverage analysis frameworks support the format out of the box, for example: + +- [Istanbul](https://istanbul.js.org/docs/advanced/alternative-reporters/#cobertura) (JavaScript) +- [Coverage.py](https://coverage.readthedocs.io/en/coverage-5.0.4/cmd.html#xml-reporting) (Python) +- [PHPUnit](https://github.com/sebastianbergmann/phpunit-documentation-english/blob/master/src/textui.rst#command-line-options) (PHP) + +Once configured, if you create a merge request that triggers a pipeline which collects +coverage reports, the coverage is shown in the diff view. This includes reports +from any job in any stage in the pipeline. The coverage displays for each line: + +- `covered` (green): lines which have been checked at least once by tests +- `no test coverage` (orange): lines which are loaded but never executed +- no coverage information: lines which are non-instrumented or not loaded + +Hovering over the coverage bar provides further information, such as the number +of times the line was checked by tests. + +Uploading a test coverage report does not enable: + +- [Test coverage results in merge requests](../pipelines/settings.md#merge-request-test-coverage-results). +- [Code coverage history](../pipelines/settings.md#view-code-coverage-history). + +You must configure these separately. + +### Limits + +A limit of 100 `<source>` nodes for Cobertura format XML files applies. If your Cobertura report exceeds +100 nodes, there can be mismatches or no matches in the merge request diff view. + +A single Cobertura XML file can be no more than 10MiB. For large projects, split the Cobertura XML into +smaller files. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328772) for more details. +When submitting many files, it can take a few minutes for coverage to show on a merge request. + +The visualization only displays after the pipeline is complete. If the pipeline has +a [blocking manual job](../jobs/job_control.md#types-of-manual-jobs), the +pipeline waits for the manual job before continuing and is not considered complete. +The visualization cannot be displayed if the blocking manual job did not run. + +### Artifact expiration + +By default, the [pipeline artifact](../pipelines/pipeline_artifacts.md#storage) used +to draw the visualization on the merge request expires **one week** after creation. + +### Coverage report from child pipeline + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363301) in GitLab 15.1 [with a flag](../../administration/feature_flags.md) named `ci_child_pipeline_coverage_reports`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/363557) and feature flag `ci_child_pipeline_coverage_reports` removed in GitLab 15.2. + +If a job in a child pipeline creates a coverage report, the report is included in +the parent pipeline's coverage report. + +```yaml +child_test_pipeline: + trigger: + include: + - local: path/to/child_pipeline_with_coverage.yml +``` + +### Automatic class path correction + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217664) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284822) in GitLab 13.9. + +The coverage report properly matches changed files only if the `filename` of a `class` element +contains the full path relative to the project root. However, in some coverage analysis frameworks, +the generated Cobertura XML has the `filename` path relative to the class package directory instead. + +To make an intelligent guess on the project root relative `class` path, the Cobertura XML parser +attempts to build the full path by: + +- Extracting a portion of the `source` paths from the `sources` element and combining them with the + class `filename` path. +- Checking if the candidate path exists in the project. +- Using the first candidate that matches as the class full path. + +#### Path correction example + +As an example, a project with: + +- A full path of `test-org/test-project`. +- The following files relative to the project root: + + ```shell + Auth/User.cs + Lib/Utils/User.cs + src/main/java + ``` + +In the: + +- Cobertura XML, the `filename` attribute in the `class` element assumes the value is a relative + path to the project's root: + + ```xml + <class name="packet.name" filename="src/main/java" line-rate="0.0" branch-rate="0.0" complexity="5"> + ``` + +- `sources` from Cobertura XML, the following paths in the format + `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`: + + ```xml + <sources> + <source>/builds/test-org/test-project/Auth</source> + <source>/builds/test-org/test-project/Lib/Utils</source> + </sources> + ``` + +The parser: + +- Extracts `Auth` and `Lib/Utils` from the `sources` and uses these to determine the `class` path + relative to the project root. +- Combines these extracted `sources` and the class filename. For example, if there is a `class` + element with the `filename` value of `User.cs`, the parser takes the first candidate path that + matches, which is `Auth/User.cs`. +- For each `class` element, attempts to look for a match for each extracted `source` path up to + 100 iterations. If it reaches this limit without finding a matching path in the file tree, the + class is not included in the final coverage report. + +NOTE: +Automatic class path correction only works on `source` paths in the format `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`. +The `source` is ignored if the path does not follow this pattern. The parser assumes that the +`filename` of a `class` element contains the full path relative to the project root. + +## Example test coverage configurations + +This section provides test coverage configuration examples for different programming languages. You can also see a working example in +the [`coverage-report`](https://gitlab.com/gitlab-org/ci-sample-projects/coverage-report/) demonstration project. + +### JavaScript example + +The following [`.gitlab-ci.yml`](../yaml/index.md) example uses [Mocha](https://mochajs.org/) +JavaScript testing and [nyc](https://github.com/istanbuljs/nyc) coverage-tooling to +generate the coverage artifact: + +```yaml +test: + script: + - npm install + - npx nyc --reporter cobertura mocha + artifacts: + reports: + coverage_report: + coverage_format: cobertura + path: coverage/cobertura-coverage.xml +``` + +### Java and Kotlin examples + +#### Maven example + +The following [`.gitlab-ci.yml`](../yaml/index.md) example for Java or Kotlin uses [Maven](https://maven.apache.org/) +to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to +generate the coverage artifact. +You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image. + +GitLab expects the artifact in the Cobertura format, so you have to execute a few +scripts before uploading it. The `test-jdk11` job tests the code and generates an +XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report: + +```yaml +test-jdk11: + stage: test + image: maven:3.6.3-jdk-11 + script: + - mvn $MAVEN_CLI_OPTS clean org.jacoco:jacoco-maven-plugin:prepare-agent test jacoco:report + artifacts: + paths: + - target/site/jacoco/jacoco.xml + +coverage-jdk11: + # Must be in a stage later than test-jdk11's stage. + # The `visualize` stage does not exist by default. + # Please define it first, or choose an existing stage like `deploy`. + stage: visualize + image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7 + script: + # convert report from jacoco to cobertura, using relative project path + - python /opt/cover2cover.py target/site/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > target/site/cobertura.xml + needs: ["test-jdk11"] + artifacts: + reports: + coverage_report: + coverage_format: cobertura + path: target/site/cobertura.xml +``` + +#### Gradle example + +The following [`.gitlab-ci.yml`](../yaml/index.md) example for Java or Kotlin uses [Gradle](https://gradle.org/) +to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to +generate the coverage artifact. +You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image. + +GitLab expects the artifact in the Cobertura format, so you have to execute a few +scripts before uploading it. The `test-jdk11` job tests the code and generates an +XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report: + +```yaml +test-jdk11: + stage: test + image: gradle:6.6.1-jdk11 + script: + - 'gradle test jacocoTestReport' # jacoco must be configured to create an xml report + artifacts: + paths: + - build/jacoco/jacoco.xml + +coverage-jdk11: + # Must be in a stage later than test-jdk11's stage. + # The `visualize` stage does not exist by default. + # Please define it first, or chose an existing stage like `deploy`. + stage: visualize + image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7 + script: + # convert report from jacoco to cobertura, using relative project path + - python /opt/cover2cover.py build/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > build/cobertura.xml + needs: ["test-jdk11"] + artifacts: + reports: + coverage_report: + coverage_format: cobertura + path: build/cobertura.xml +``` + +### Python example + +The following [`.gitlab-ci.yml`](../yaml/index.md) example for Python uses [pytest-cov](https://pytest-cov.readthedocs.io/) to collect test coverage data and [coverage.py](https://coverage.readthedocs.io/) to convert the report to use full relative paths. +The information isn't displayed without the conversion. + +This example assumes that the code for your package is in `src/` and your tests are in `tests.py`: + +```yaml +run tests: + stage: test + image: python:3 + script: + - pip install pytest pytest-cov + - coverage run -m pytest + - coverage report + - coverage xml + coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/' + artifacts: + reports: + coverage_report: + coverage_format: cobertura + path: coverage.xml +``` + +### PHP example + +The following [`.gitlab-ci.yml`](../yaml/index.md) example for PHP uses [PHPUnit](https://phpunit.readthedocs.io/) +to collect test coverage data and generate the report. + +With a minimal [`phpunit.xml`](https://phpunit.readthedocs.io/en/9.5/configuration.html) file (you may reference +[this example repository](https://gitlab.com/yookoala/code-coverage-visualization-with-php/)), you can run the test and +generate the `coverage.xml`: + +```yaml +run tests: + stage: test + image: php:latest + variables: + XDEBUG_MODE: coverage + before_script: + - apt-get update && apt-get -yq install git unzip zip libzip-dev zlib1g-dev + - docker-php-ext-install zip + - pecl install xdebug && docker-php-ext-enable xdebug + - php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" + - php composer-setup.php --install-dir=/usr/local/bin --filename=composer + - composer install + - composer require --dev phpunit/phpunit phpunit/php-code-coverage + script: + - php ./vendor/bin/phpunit --coverage-text --coverage-cobertura=coverage.cobertura.xml + artifacts: + reports: + coverage_report: + coverage_format: cobertura + path: coverage.cobertura.xml +``` + +[Codeception](https://codeception.com/), through PHPUnit, also supports generating Cobertura report with +[`run`](https://codeception.com/docs/reference/Commands#run). The path for the generated file +depends on the `--coverage-cobertura` option and [`paths`](https://codeception.com/docs/reference/Configuration#paths) +configuration for the [unit test suite](https://codeception.com/docs/05-UnitTests). Configure `.gitlab-ci.yml` +to find Cobertura in the appropriate path. + +### C/C++ example + +The following [`.gitlab-ci.yml`](../yaml/index.md) example for C/C++ with +`gcc` or `g++` as the compiler uses [`gcovr`](https://gcovr.com/en/stable/) to generate the coverage +output file in Cobertura XML format. + +This example assumes: + +- That the `Makefile` is created by `cmake` in the `build` directory, + within another job in a previous stage. + (If you use `automake` to generate the `Makefile`, + then you need to call `make check` instead of `make test`.) +- `cmake` (or `automake`) has set the compiler option `--coverage`. + +```yaml +run tests: + stage: test + script: + - cd build + - make test + - gcovr --xml-pretty --exclude-unreachable-branches --print-summary -o coverage.xml --root ${CI_PROJECT_DIR} + coverage: /^\s*lines:\s*\d+.\d+\%/ + artifacts: + name: ${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHA} + expire_in: 2 days + reports: + coverage_report: + coverage_format: cobertura + path: build/coverage.xml +``` + +### Go example + +The following [`.gitlab-ci.yml`](../yaml/index.md) example for Go uses: + +- [`go test`](https://go.dev/doc/tutorial/add-a-test) to run tests. +- [`gocover-cobertura`](https://github.com/boumenot/gocover-cobertura) to convert Go's coverage profile into the Cobertura XML format. + +This example assumes that [Go modules](https://go.dev/ref/mod) +are being used. Please note that the `-covermode count` option does not work with the `-race` flag. +If you want to generate code coverage while also using the `-race` flag, you must switch to +`-covermode atomic` which is slower than `-covermode count`. See [this blog post](https://go.dev/blog/cover) +for more details. + +```yaml +run tests: + stage: test + image: golang:1.17 + script: + - go install + - go test ./... -coverprofile=coverage.txt -covermode count + - go get github.com/boumenot/gocover-cobertura + - go run github.com/boumenot/gocover-cobertura < coverage.txt > coverage.xml + artifacts: + reports: + coverage_report: + coverage_format: cobertura + path: coverage.xml +``` + +### Ruby example + +The following [`.gitlab-ci.yml`](../yaml/index.md) example for Ruby uses + +- [`rspec`](https://rspec.info/) to run tests. +- [`simplecov`](https://github.com/simplecov-ruby/simplecov) and [`simplecov-cobertura`](https://github.com/dashingrocket/simplecov-cobertura) + to record the coverage profile and create a report in the Cobertura XML format. + +This example assumes: + +- That [`bundler`](https://bundler.io/) is being used for dependency management. + The `rspec`, `simplecov` and `simplecov-cobertura` gems have been added to your `Gemfile`. +- The `CoberturaFormatter` has been added to your `SimpleCov.formatters` + configuration within the `spec_helper.rb` file. + +```yaml +run tests: + stage: test + image: ruby:3.1 + script: + - bundle install + - bundle exec rspec + artifacts: + reports: + coverage_report: + coverage_format: cobertura + path: coverage/coverage.xml +``` + +## Troubleshooting + +### Test coverage visualization not displayed + +If the test coverage visualization is not displayed in the diff view, you can check +the coverage report itself and verify that: + +- The file you are viewing in the diff view is mentioned in the coverage report. +- The `source` and `filename` nodes in the report follows the [expected structure](#automatic-class-path-correction) + to match the files in your repository. + +Report artifacts are not downloadable by default. If you want the report to be downloadable +from the job details page, add your coverage report to the artifact `paths`: + +```yaml +artifacts: + paths: + - coverage/cobertura-coverage.xml + reports: + coverage_report: + coverage_format: cobertura + path: coverage/cobertura-coverage.xml +``` diff --git a/doc/ci/testing/unit_test_reports.md b/doc/ci/testing/unit_test_reports.md index 8aa41cd6fc0..c8e0d6135df 100644 --- a/doc/ci/testing/unit_test_reports.md +++ b/doc/ci/testing/unit_test_reports.md @@ -145,8 +145,9 @@ GitLab does not parse very [large nodes](https://nokogiri.org/tutorials/parsing_ > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202114) in GitLab 13.0 behind the `:junit_pipeline_screenshots_view` feature flag, disabled by default. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/216979) in GitLab 13.12. -Upload your screenshots as [artifacts](../yaml/artifacts_reports.md#artifactsreportsjunit) to GitLab. If JUnit -report format XML files contain an `attachment` tag, GitLab parses the attachment. Note that: +You can upload your screenshots as [artifacts](../yaml/artifacts_reports.md#artifactsreportsjunit) to GitLab. +If JUnit report format XML files contain an `attachment` tag, GitLab parses the attachment. +When uploading screenshot artifacts: - The `attachment` tag **must** contain the relative path to `$CI_PROJECT_DIR` of the screenshots you uploaded. For example: @@ -161,5 +162,18 @@ report format XML files contain an `attachment` tag, GitLab parses the attachmen [`artifacts:when: always`](../yaml/index.md#artifactswhen) so that it still uploads a screenshot when a test fails. -A link to the test case attachment appears in the test case details in -[the pipeline test report](#view-unit-test-reports-on-gitlab). +After the attachment is uploaded, [the pipeline test report](#view-unit-test-reports-on-gitlab) +contains a link to the screenshot, for example: + +![Unit test report screenshot example](img/unit_test_report_screenshot_v13_12.png) + +## Troubleshooting + +### Test report appears empty + +A unit test report can appear to be empty when [viewed in a merge request](#view-unit-test-reports-on-gitlab) +if the artifact that contained the report [expires](../yaml/index.md#artifactsexpire_in). +If the artifact frequently expires too early, set a longer `expire_in` value for +the report artifact. + +Alternatively, you can run a new pipeline to generate a new report. diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 8d8afbffab9..0230aaf7113 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -361,6 +361,29 @@ When you visit the job log page for a running job, there could be a delay of up 60 seconds before the log updates. The default refresh time is 60 seconds, but after the log is viewed in the UI, the following log updates should occur every 3 seconds. +## Disaster recovery + +You can disable some important but computationally expensive parts of the application +to relieve stress on the database during ongoing downtime. + +### Disable fair scheduling on shared runners + +When clearing a large backlog of jobs, you can temporarily enable the `ci_queueing_disaster_recovery_disable_fair_scheduling` +[feature flag](../administration/feature_flags.md). This flag disables fair scheduling +on shared runners, which reduces system resource usage on the `jobs/request` endpoint. + +When enabled, jobs are processed in the order they were put in the system, instead of +balanced across many projects. + +### Disable CI/CD minutes quota enforcement + +To disable the enforcement of CI/CD minutes quotas on shared runners, you can temporarily +enable the `ci_queueing_disaster_recovery_disable_quota` [feature flag](../administration/feature_flags.md). +This flag reduces system resource usage on the `jobs/request` endpoint. + +When enabled, jobs created in the last hour can run in projects which are out of quota. +Earlier jobs are already canceled by a periodic background worker (`StuckCiJobsWorker`). + ## How to get help If you are unable to resolve pipeline issues, you can get help from: diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index c53fad69376..72df8d56815 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -346,7 +346,7 @@ To mask a variable: 1. Select the **Mask variable** checkbox. 1. Select **Update variable**. -The method used to mask variables [limits what can be included in a masked variable](](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13784#note_106756757)). +The method used to mask variables [limits what can be included in a masked variable](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/13784#note_106756757). The value of the variable must: - Be a single line. @@ -401,6 +401,8 @@ Review all merge requests that introduce changes to the `.gitlab-ci.yml` file be - [Run a pipeline in the parent project for a merge request submitted from a forked project](../pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project). - Merge the changes. +Review the `.gitlab-ci.yml` file of imported projects before you add files or run pipelines against them. + The following example shows malicious code in a `.gitlab-ci.yml` file: ```yaml diff --git a/doc/ci/variables/predefined_variables.md b/doc/ci/variables/predefined_variables.md index e6d61ef342b..6df614c1cda 100644 --- a/doc/ci/variables/predefined_variables.md +++ b/doc/ci/variables/predefined_variables.md @@ -29,7 +29,7 @@ as it can cause the pipeline to behave unexpectedly. | `CI_API_V4_URL` | 11.7 | all | The GitLab API v4 root URL. | | `CI_BUILDS_DIR` | all | 11.10 | The top-level directory where builds are executed. | | `CI_COMMIT_AUTHOR` | 13.11 | all | The author of the commit in `Name <email>` format. | -| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch. Is always `0000000000000000000000000000000000000000` in merge request pipelines. | +| `CI_COMMIT_BEFORE_SHA` | 11.2 | all | The previous latest commit present on a branch or tag. Is always `0000000000000000000000000000000000000000` in merge request pipelines and for the first commit in pipelines for branches or tags. | | `CI_COMMIT_BRANCH` | 12.6 | 0.5 | The commit branch name. Available in branch pipelines, including pipelines for the default branch. Not available in merge request pipelines or tag pipelines. | | `CI_COMMIT_DESCRIPTION` | 10.8 | all | The description of the commit. If the title is shorter than 100 characters, the message without the first line. | | `CI_COMMIT_MESSAGE` | 10.8 | all | The full commit message. | @@ -139,29 +139,30 @@ These variables are available when: - The pipelines [are merge request pipelines](../pipelines/merge_request_pipelines.md). - The merge request is open. -| Variable | GitLab | Runner | Description | -|----------------------------------------|--------|--------|-------------| -| `CI_MERGE_REQUEST_APPROVED` | 14.1 | all | Approval status of the merge request. `true` when [merge request approvals](../../user/project/merge_requests/approvals/index.md) is available and the merge request has been approved. | -| `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of usernames of assignees for the merge request. | -| `CI_MERGE_REQUEST_ID` | 11.6 | all | The instance-level ID of the merge request. This is a unique ID across all projects on GitLab. | -| `CI_MERGE_REQUEST_IID` | 11.6 | all | The project-level IID (internal ID) of the merge request. This ID is unique for the current project. | -| `CI_MERGE_REQUEST_LABELS` | 11.9 | all | Comma-separated label names of the merge request. | -| `CI_MERGE_REQUEST_MILESTONE` | 11.9 | all | The milestone title of the merge request. | -| `CI_MERGE_REQUEST_PROJECT_ID` | 11.6 | all | The ID of the project of the merge request. | -| `CI_MERGE_REQUEST_PROJECT_PATH` | 11.6 | all | The path of the project of the merge request. For example `namespace/awesome-project`. | -| `CI_MERGE_REQUEST_PROJECT_URL` | 11.6 | all | The URL of the project of the merge request. For example, `http://192.168.10.15:3000/namespace/awesome-project`. | -| `CI_MERGE_REQUEST_REF_PATH` | 11.6 | all | The ref path of the merge request. For example, `refs/merge-requests/1/head`. | -| `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | 11.6 | all | The source branch name of the merge request. | -| `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the source branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/merged_results_pipelines.md). | -| `CI_MERGE_REQUEST_SOURCE_PROJECT_ID` | 11.6 | all | The ID of the source project of the merge request. | -| `CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` | 11.6 | all | The path of the source project of the merge request. | -| `CI_MERGE_REQUEST_SOURCE_PROJECT_URL` | 11.6 | all | The URL of the source project of the merge request. | -| `CI_MERGE_REQUEST_TARGET_BRANCH_NAME` | 11.6 | all | The target branch name of the merge request. | -| `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the target branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/merged_results_pipelines.md). | -| `CI_MERGE_REQUEST_TITLE` | 11.9 | all | The title of the merge request. | -| `CI_MERGE_REQUEST_EVENT_TYPE` | 12.3 | all | The event type of the merge request. Can be `detached`, `merged_result` or `merge_train`. | -| `CI_MERGE_REQUEST_DIFF_ID` | 13.7 | all | The version of the merge request diff. | -| `CI_MERGE_REQUEST_DIFF_BASE_SHA` | 13.7 | all | The base SHA of the merge request diff. | +| Variable | GitLab | Runner | Description | +|---------------------------------------------|--------|--------|-------------| +| `CI_MERGE_REQUEST_APPROVED` | 14.1 | all | Approval status of the merge request. `true` when [merge request approvals](../../user/project/merge_requests/approvals/index.md) is available and the merge request has been approved. | +| `CI_MERGE_REQUEST_ASSIGNEES` | 11.9 | all | Comma-separated list of usernames of assignees for the merge request. | +| `CI_MERGE_REQUEST_ID` | 11.6 | all | The instance-level ID of the merge request. This is a unique ID across all projects on GitLab. | +| `CI_MERGE_REQUEST_IID` | 11.6 | all | The project-level IID (internal ID) of the merge request. This ID is unique for the current project. | +| `CI_MERGE_REQUEST_LABELS` | 11.9 | all | Comma-separated label names of the merge request. | +| `CI_MERGE_REQUEST_MILESTONE` | 11.9 | all | The milestone title of the merge request. | +| `CI_MERGE_REQUEST_PROJECT_ID` | 11.6 | all | The ID of the project of the merge request. | +| `CI_MERGE_REQUEST_PROJECT_PATH` | 11.6 | all | The path of the project of the merge request. For example `namespace/awesome-project`. | +| `CI_MERGE_REQUEST_PROJECT_URL` | 11.6 | all | The URL of the project of the merge request. For example, `http://192.168.10.15:3000/namespace/awesome-project`. | +| `CI_MERGE_REQUEST_REF_PATH` | 11.6 | all | The ref path of the merge request. For example, `refs/merge-requests/1/head`. | +| `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | 11.6 | all | The source branch name of the merge request. | +| `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the source branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/merged_results_pipelines.md). | +| `CI_MERGE_REQUEST_SOURCE_PROJECT_ID` | 11.6 | all | The ID of the source project of the merge request. | +| `CI_MERGE_REQUEST_SOURCE_PROJECT_PATH` | 11.6 | all | The path of the source project of the merge request. | +| `CI_MERGE_REQUEST_SOURCE_PROJECT_URL` | 11.6 | all | The URL of the source project of the merge request. | +| `CI_MERGE_REQUEST_TARGET_BRANCH_NAME` | 11.6 | all | The target branch name of the merge request. | +| `CI_MERGE_REQUEST_TARGET_BRANCH_PROTECTED` | 15.2 | all | The protection status for the target branch of the merge request. | +| `CI_MERGE_REQUEST_TARGET_BRANCH_SHA` | 11.9 | all | The HEAD SHA of the target branch of the merge request. The variable is empty in merge request pipelines. The SHA is present only in [merged results pipelines](../pipelines/merged_results_pipelines.md). | +| `CI_MERGE_REQUEST_TITLE` | 11.9 | all | The title of the merge request. | +| `CI_MERGE_REQUEST_EVENT_TYPE` | 12.3 | all | The event type of the merge request. Can be `detached`, `merged_result` or `merge_train`. | +| `CI_MERGE_REQUEST_DIFF_ID` | 13.7 | all | The version of the merge request diff. | +| `CI_MERGE_REQUEST_DIFF_BASE_SHA` | 13.7 | all | The base SHA of the merge request diff. | ## Predefined variables for external pull request pipelines diff --git a/doc/ci/yaml/artifacts_reports.md b/doc/ci/yaml/artifacts_reports.md index e4324ab06e1..379a4b3224a 100644 --- a/doc/ci/yaml/artifacts_reports.md +++ b/doc/ci/yaml/artifacts_reports.md @@ -36,9 +36,9 @@ The `accessibility` report uses [pa11y](https://pa11y.org/) to report on the acc of changes introduced in merge requests. GitLab can display the results of one or more reports in the merge request -[accessibility widget](../../user/project/merge_requests/accessibility_testing.md#accessibility-merge-request-widget). +[accessibility widget](../testing/accessibility_testing.md#accessibility-merge-request-widget). -For more information, see [Accessibility testing](../../user/project/merge_requests/accessibility_testing.md). +For more information, see [Accessibility testing](../testing/accessibility_testing.md). ## `artifacts:reports:api_fuzzing` **(ULTIMATE)** @@ -52,18 +52,18 @@ GitLab can display the results of one or more reports in: - The merge request [security widget](../../user/application_security/api_fuzzing/index.md#view-details-of-an-api-fuzzing-vulnerability). - The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#view-vulnerabilities-in-a-pipeline). +- The pipeline [**Security** tab](../../user/application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline). - The [security dashboard](../../user/application_security/api_fuzzing/index.md#security-dashboard). ## `artifacts:reports:browser_performance` **(PREMIUM)** > [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. -The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) +The `browser_performance` report collects [Browser Performance Testing metrics](../testing/browser_performance_testing.md) as artifacts. GitLab can display the results of one report in the merge request -[browser performance testing widget](../../user/project/merge_requests/browser_performance_testing.md#how-browser-performance-testing-works). +[browser performance testing widget](../testing/browser_performance_testing.md#how-browser-performance-testing-works). GitLab cannot display the combined results of multiple `browser_performance` reports. @@ -100,7 +100,7 @@ instead. Use `coverage_report` to collect coverage report in Cobertura format. -The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). +The `cobertura` report collects [Cobertura coverage XML files](../testing/test_coverage_visualization.md). Cobertura was originally developed for Java, but there are many third-party ports for other languages such as JavaScript, Python, and Ruby. @@ -116,22 +116,22 @@ artifacts: The collected coverage report is uploaded to GitLab as an artifact. GitLab can display the results of coverage report in the merge request -[diff annotations](../../user/project/merge_requests/test_coverage_visualization.md). +[diff annotations](../testing/test_coverage_visualization.md). ## `artifacts:reports:codequality` > [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. -The `codequality` report collects [code quality issues](../../user/project/merge_requests/code_quality.md). The +The `codequality` report collects [code quality issues](../testing/code_quality.md). The collected code quality report uploads to GitLab as an artifact. GitLab can display the results of: -- One or more reports in the merge request [code quality widget](../../user/project/merge_requests/code_quality.md#code-quality-widget). +- One or more reports in the merge request [code quality widget](../testing/code_quality.md#code-quality-widget). - Only one report in: - - The merge request [diff annotations](../../user/project/merge_requests/code_quality.md#code-quality-in-diff-view). + - The merge request [diff annotations](../testing/code_quality.md#code-quality-in-diff-view). Track progress on adding support for multiple reports in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328257). - - The [full report](../metrics_reports.md). Track progress on adding support for multiple reports in + - The [full report](../testing/metrics_reports.md). Track progress on adding support for multiple reports in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/9014). ## `artifacts:reports:container_scanning` **(ULTIMATE)** @@ -142,7 +142,7 @@ The collected Container Scanning report uploads to GitLab as an artifact. GitLab can display the results of one or more reports in: - The merge request [container scanning widget](../../user/application_security/container_scanning/index.md). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#view-vulnerabilities-in-a-pipeline). +- The pipeline [**Security** tab](../../user/application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline). - The [security dashboard](../../user/application_security/security_dashboard/index.md). - The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). @@ -156,7 +156,7 @@ The collected coverage fuzzing report uploads to GitLab as an artifact. GitLab can display the results of one or more reports in: - The merge request [coverage fuzzing widget](../../user/application_security/coverage_fuzzing/index.md#interacting-with-the-vulnerabilities). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#view-vulnerabilities-in-a-pipeline). +- The pipeline [**Security** tab](../../user/application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline). - The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). - The [security dashboard](../../user/application_security/security_dashboard/index.md). @@ -168,7 +168,7 @@ report uploads to GitLab as an artifact. GitLab can display the results of one or more reports in: - The merge request [security widget](../../user/application_security/dast/index.md#view-details-of-a-vulnerability-detected-by-dast). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#view-vulnerabilities-in-a-pipeline). +- The pipeline [**Security** tab](../../user/application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline). - The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). - The [security dashboard](../../user/application_security/security_dashboard/index.md). @@ -180,7 +180,7 @@ The collected Dependency Scanning report uploads to GitLab as an artifact. GitLab can display the results of one or more reports in: - The merge request [dependency scanning widget](../../user/application_security/dependency_scanning/index.md). -- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#view-vulnerabilities-in-a-pipeline). +- The pipeline [**Security** tab](../../user/application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline). - The [security dashboard](../../user/application_security/security_dashboard/index.md). - The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). - The [dependency list](../../user/application_security/dependency_list/). @@ -213,6 +213,7 @@ The exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv - The `.env` file can't have empty lines or comments (starting with `#`). - Key values in the `env` file cannot have spaces or newline characters (`\n`), including when using single or double quotes. - Quote escaping during parsing (`key = 'value'` -> `{key: "value"}`) is not supported. +- Only UTF-8 encoding is [supported](../pipelines/job_artifacts.md#error-message-fatal-invalid-argument-when-uploading-a-dotenv-artifact-on-a-windows-runner). ## `artifacts:reports:junit` @@ -263,25 +264,25 @@ GitLab can display the results of one or more reports in: > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in GitLab 13.2. > - Requires GitLab Runner 11.5 and above. -The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md). +The `load_performance` report collects [Load Performance Testing metrics](../testing/load_performance_testing.md). The report is uploaded to GitLab as an artifact. GitLab can display the results of only one report in the merge request -[load testing widget](../../user/project/merge_requests/load_performance_testing.md#how-load-performance-testing-works). +[load testing widget](../testing/load_performance_testing.md#how-load-performance-testing-works). GitLab cannot display the combined results of multiple `load_performance` reports. ## `artifacts:reports:metrics` **(PREMIUM)** -The `metrics` report collects [Metrics](../metrics_reports.md). The collected Metrics report uploads to GitLab as an +The `metrics` report collects [Metrics](../testing/metrics_reports.md). The collected Metrics report uploads to GitLab as an artifact. GitLab can display the results of one or more reports in the merge request -[metrics reports widget](../../ci/metrics_reports.md#metrics-reports). +[metrics reports widget](../testing/metrics_reports.md#metrics-reports). ## `artifacts:reports:requirements` **(ULTIMATE)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. The `requirements` report collects `requirements.json` files. The collected Requirements report uploads to GitLab as an artifact and existing [requirements](../../user/project/requirements/index.md) are marked as Satisfied. @@ -291,7 +292,7 @@ GitLab can display the results of one or more reports in the ## `artifacts:reports:sast` -> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) from GitLab Ultimate to GitLab Free in 13.3. +> [Moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) from GitLab Ultimate to GitLab Free in 13.3. The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md). The collected SAST report uploads to GitLab as an artifact. diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index 912eca364c9..3bb2007d6e3 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -675,6 +675,45 @@ artifacts are restored after [caches](#cache). [Read more about artifacts](../pipelines/job_artifacts.md). +#### `artifacts:paths` + +Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly +link outside it. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default` section](#default). + +**Possible inputs**: + +- An array of file paths, relative to the project directory. +- You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) + patterns and: + - In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), + [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). + - In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). + +**Example of `artifacts:paths`**: + +```yaml +job: + artifacts: + paths: + - binaries/ + - .config +``` + +This example creates an artifact with `.config` and all the files in the `binaries` directory. + +**Additional details**: + +- If not used with [`artifacts:name`](#artifactsname), the artifacts file + is named `artifacts`, which becomes `artifacts.zip` when downloaded. + +**Related topics**: + +- To restrict which jobs a specific job fetches artifacts from, see [`dependencies`](#dependencies). +- [Create job artifacts](../pipelines/job_artifacts.md#create-job-artifacts). + #### `artifacts:exclude` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15122) in GitLab 13.1 @@ -843,45 +882,6 @@ job: - [Use CI/CD variables to define the artifacts name](../pipelines/job_artifacts.md#use-cicd-variables-to-define-the-artifacts-name). -#### `artifacts:paths` - -Paths are relative to the project directory (`$CI_PROJECT_DIR`) and can't directly -link outside it. - -**Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default` section](#default). - -**Possible inputs**: - -- An array of file paths, relative to the project directory. -- You can use Wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) - patterns and: - - In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), - [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). - - In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). - -**Example of `artifacts:paths`**: - -```yaml -job: - artifacts: - paths: - - binaries/ - - .config -``` - -This example creates an artifact with `.config` and all the files in the `binaries` directory. - -**Additional details**: - -- If not used with [`artifacts:name`](#artifactsname) defined, the artifacts file - is named `artifacts`, which becomes `artifacts.zip` when downloaded. - -**Related topics**: - -- To restrict which jobs a specific job fetches artifacts from, see [`dependencies`](#dependencies). -- [Create job artifacts](../pipelines/job_artifacts.md#create-job-artifacts). - #### `artifacts:public` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/49775) in GitLab 13.8 @@ -910,7 +910,7 @@ pipelines, set `artifacts:public` to `false`: - `true` (default if not defined) or `false`. -**Example of `artifacts:paths`**: +**Example of `artifacts:public`**: ```yaml job: @@ -1099,6 +1099,8 @@ that use the same cache key use the same cache, including in different pipelines If not set, the default key is `default`. All jobs with the `cache` keyword but no `cache:key` share the `default` cache. +Must be used with `cache: path`, or nothing is cached. + **Keyword type**: Job keyword. You can use it only as part of a job or in the [`default` section](#default). @@ -1263,6 +1265,8 @@ rspec: Use `cache:when` to define when to save the cache, based on the status of the job. +Must be used with `cache: path`, or nothing is cached. + **Keyword type**: Job keyword. You can use it only as part of a job or in the [`default` section](#default). @@ -1301,6 +1305,8 @@ Use the `pull` policy when you have many jobs executing in parallel that use the This policy speeds up job execution and reduces load on the cache server. You can use a job with the `push` policy to build the cache. +Must be used with `cache: path`, or nothing is cached. + **Keyword type**: Job keyword. You can use it only as part of a job or in the [`default` section](#default). @@ -2204,6 +2210,7 @@ In this example: Use `needs:project` to download artifacts from up to five jobs in other pipelines. The artifacts are downloaded from the latest successful pipeline for the specified ref. +To specify multiple jobs, add each as separate array items under the `needs` keyword. If there is a pipeline running for the specified ref, a job with `needs:project` does not wait for the pipeline to complete. Instead, the job downloads the artifact @@ -2232,10 +2239,14 @@ build_job: job: build-1 ref: main artifacts: true + - project: namespace/group/project-name-2 + job: build-2 + ref: main + artifacts: true ``` -In this example, `build_job` downloads the artifacts from the latest successful `build-1` job -on the `main` branch in the `group/project-name` project. +In this example, `build_job` downloads the artifacts from the latest successful `build-1` and `build-2` jobs +on the `main` branches in the `group/project-name` and `group/project-name-2` projects. In GitLab 13.3 and later, you can use [CI/CD variables](../variables/where_variables_can_be_used.md#gitlab-ciyml-file) in `needs:project`, for example: @@ -2428,13 +2439,6 @@ You can use `only` and `except` to control when to add jobs to pipelines. - Use `only` to define when a job runs. - Use `except` to define when a job **does not** run. -Four keywords can be used with `only` and `except`: - -- [`refs`](#onlyrefs--exceptrefs) -- [`variables`](#onlyvariables--exceptvariables) -- [`changes`](#onlychanges--exceptchanges) -- [`kubernetes`](#onlykubernetes--exceptkubernetes) - See [specify when jobs run with `only` and `except`](../jobs/job_control.md#specify-when-jobs-run-with-only-and-except) for more details and examples. @@ -2443,6 +2447,10 @@ for more details and examples. Use the `only:refs` and `except:refs` keywords to control when to add jobs to a pipeline based on branch names or pipeline types. +`only:refs` and `except:refs` are not being actively developed. [`rules:if`](#rulesif) +is the preferred keyword when using refs, regular expressions, or variables to control +when to add jobs to pipelines. + **Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: An array including any number of: @@ -2519,8 +2527,8 @@ job2: job2: script: echo "test" only: - - branches - - tags + - branches + - tags ``` #### `only:variables` / `except:variables` @@ -2528,6 +2536,10 @@ job2: Use the `only:variables` or `except:variables` keywords to control when to add jobs to a pipeline, based on the status of [CI/CD variables](../variables/index.md). +`only:variables` and `except:variables` are not being actively developed. [`rules:if`](#rulesif) +is the preferred keyword when using refs, regular expressions, or variables to control +when to add jobs to pipelines. + **Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: @@ -2560,6 +2572,9 @@ Use `changes` in pipelines with the following refs: - `external_pull_requests` - `merge_requests` (see additional details about [using `only:changes` with merge request pipelines](../jobs/job_control.md#use-onlychanges-with-merge-request-pipelines)) +`only:changes` and `except:changes` are not being actively developed. [`rules:changes`](#ruleschanges) +is the preferred keyword when using changed files to control when to add jobs to pipelines. + **Keyword type**: Job keyword. You can use it only as part of a job. **Possible inputs**: An array including any number of: @@ -2610,6 +2625,10 @@ docker build: Use `only:kubernetes` or `except:kubernetes` to control if jobs are added to the pipeline when the Kubernetes service is active in the project. +`only:refs` and `except:refs` are not being actively developed. Use [`rules:if`](#rulesif) +with the [`CI_KUBERNETES_ACTIVE`](../variables/predefined_variables.md) predefined CI/CD variable +to control if jobs are added to the pipeline when the Kubernetes service is active in the project. + **Keyword type**: Job-specific. You can use it only as part of a job. **Possible inputs**: @@ -3388,7 +3407,7 @@ This keyword must be used with `secrets:vault`. #### `secrets:vault` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in GitLab 13.4 and GitLab Runner 13.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in GitLab 13.4 and GitLab Runner 13.4. Use `secrets:vault` to specify secrets provided by a [HashiCorp Vault](https://www.vaultproject.io/). @@ -3519,6 +3538,52 @@ in that container. - [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). - [Use Docker to build Docker images](../docker/using_docker_build.md). +#### `service:pull_policy` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21619) in GitLab 15.1 [with a flag](../../administration/feature_flags.md) named `ci_docker_image_pull_policy`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/363186) in GitLab 15.2. +> - Requires GitLab Runner 15.1 or later. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, +ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `ci_docker_image_pull_policy`. + +The pull policy that the runner uses to fetch the Docker image. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the [`default` section](#default). + +**Possible inputs**: + +- A single pull policy, or multiple pull policies in an array. + Can be `always`, `if-not-present`, or `never`. + +**Examples of `service:pull_policy`**: + +```yaml +job1: + script: echo "A single pull policy." + services: + - name: postgres:11.6 + pull_policy: if-not-present + +job2: + script: echo "Multiple pull policies." + services: + - name: postgres:11.6 + pull_policy: [always, if-not-present] +``` + +**Additional details**: + +- If the runner does not support the defined pull policy, the job fails with an error similar to: + `ERROR: Job failed (system failure): the configured PullPolicies ([always]) are not allowed by AllowedPullPolicies ([never])`. + +**Related topics**: + +- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). +- [How runner pull policies work](https://docs.gitlab.com/runner/executors/docker.html#how-pull-policies-work). +- [Using multiple pull policies](https://docs.gitlab.com/runner/executors/docker.html#using-multiple-pull-policies). + ### `stage` Use `stage` to define which [stage](#stages) a job runs in. Jobs in the same diff --git a/doc/ci/yaml/workflow.md b/doc/ci/yaml/workflow.md index a985db14d08..743a2639c0c 100644 --- a/doc/ci/yaml/workflow.md +++ b/doc/ci/yaml/workflow.md @@ -61,7 +61,7 @@ This example prevents pipelines for schedules or `push` (branches and tags) pipe The final `when: always` rule runs all other pipeline types, **including** merge request pipelines. -## Switch between branch pipelines and merge request pipelines +### Switch between branch pipelines and merge request pipelines > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) in GitLab 13.8. @@ -115,6 +115,25 @@ set and could be blocked by a similar rule. Triggered pipelines have a pipeline of `trigger` or `pipeline`, so `&& $CI_PIPELINE_SOURCE == "push"` ensures the rule does not block triggered pipelines. +### Git Flow with merge request pipelines + +You can use `workflow: rules` as part of [Git Flow or similar strategies](../../topics/gitlab_flow.md) +with merge request pipelines. With these rules, you can use [merge request pipeline features](../pipelines/merge_request_pipelines.md) +with feature branches, while keeping long-lived branches to support multiple versions +of your software. + +For example, to only run pipelines for your merge requests, tags, and protected branches: + +```yaml +workflow: + rules: + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_TAG + - if: $CI_COMMIT_REF_PROTECTED +``` + +This example assumes that your long-lived branches are [protected](../../user/project/protected_branches.md). + ## `workflow:rules` templates > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217732) in GitLab 13.0. @@ -150,3 +169,23 @@ To [include](index.md#include) it: include: - template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml' ``` + +## Troubleshooting + +### Merge request stuck with `Checking pipeline status.` message + +If a merge request displays `Checking pipeline status.`, but the message never goes +away (the "spinner" never stops spinning), it might be due to `workflow:rules`. +This issue can happen if a project has [**Pipelines must succeed**](../../user/project/merge_requests/merge_when_pipeline_succeeds.md#only-allow-merge-requests-to-be-merged-if-the-pipeline-succeeds) +enabled, but the `workflow:rules` prevent a pipeline from running for the merge request. + +For example, with this workflow, merge requests cannot be merged, because no +pipeline can run: + +```yaml +workflow: + rules: + - changes: + - .gitlab/**/**.md + when: never +``` diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index f524b04c6eb..e80bffe7c18 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -20,27 +20,24 @@ WHERE user_id = 2; Here we are filtering by the `user_id` column and as such a developer may decide to index this column. -While in certain cases indexing columns using the above approach may make sense -it can actually have a negative impact. Whenever you write data to a table any -existing indexes need to be updated. The more indexes there are the slower this -can potentially become. Indexes can also take up quite some disk space depending +While in certain cases indexing columns using the above approach may make sense, +it can actually have a negative impact. Whenever you write data to a table, any +existing indexes must also be updated. The more indexes there are, the slower this +can potentially become. Indexes can also take up significant disk space, depending on the amount of data indexed and the index type. For example, PostgreSQL offers -"GIN" indexes which can be used to index certain data types that can not be -indexed by regular B-tree indexes. These indexes however generally take up more +`GIN` indexes which can be used to index certain data types that cannot be +indexed by regular B-tree indexes. These indexes, however, generally take up more data and are slower to update compared to B-tree indexes. -Because of all this one should not blindly add a new index for every column used -to filter data by. Instead one should ask themselves the following questions: +Because of all this, it's important make the following considerations +when adding a new index: -1. Can you write your query in such a way that it re-uses as many existing indexes - as possible? -1. Is the data large enough that using an index is actually - faster than just iterating over the rows in the table? +1. Do the new queries re-use as many existing indexes as possible? +1. Is there enough data that using an index is faster than iterating over + rows in the table? 1. Is the overhead of maintaining the index worth the reduction in query timings? -We explore every question in detail below. - ## Re-using Queries The first step is to make sure your query re-uses as many existing indexes as @@ -59,10 +56,8 @@ unindexed. In reality the query may perform just fine given the index on `user_id` can filter out enough rows. The best way to determine if indexes are re-used is to run your query using -`EXPLAIN ANALYZE`. Depending on any extra tables that may be joined and -other columns being used for filtering you may find an extra index is not going -to make much (if any) difference. On the other hand you may determine that the -index _may_ make a difference. +`EXPLAIN ANALYZE`. Depending on the joined tables and the columns being used for filtering, +you may find an extra index doesn't make much, if any, difference. In short: @@ -73,28 +68,24 @@ In short: ## Data Size -A database may decide not to use an index despite it existing in case a regular -sequence scan (= simply iterating over all existing rows) is faster. This is -especially the case for small tables. +A database may not use an index even when a regular sequence scan +(iterating over all rows) is faster, especially for small tables. -If a table is expected to grow in size and you expect your query has to filter -out a lot of rows you may want to consider adding an index. If the table size is -very small (for example, fewer than `1,000` records) or any existing indexes filter out -enough rows you may _not_ want to add a new index. +Consider adding an index if a table is expected to grow, and your query has to filter a lot of rows. +You may _not_ want to add an index if the table size is small (<`1,000` records), +or if existing indexes already filter out enough rows. ## Maintenance Overhead -Indexes have to be updated on every table write. In case of PostgreSQL _all_ +Indexes have to be updated on every table write. In the case of PostgreSQL, _all_ existing indexes are updated whenever data is written to a table. As a -result of this having many indexes on the same table slows down writes. - -Because of this one should ask themselves: is the reduction in query performance -worth the overhead of maintaining an extra index? +result, having many indexes on the same table slows down writes. It's therefore important +to balance query performance with the overhead of maintaining an extra index. -If adding an index reduces SELECT timings by 5 milliseconds but increases -INSERT/UPDATE/DELETE timings by 10 milliseconds then the index may not be worth -it. On the other hand, if SELECT timings are reduced but INSERT/UPDATE/DELETE -timings are not affected you may want to add the index after all. +Let's say that adding an index reduces SELECT timings by 5 milliseconds but increases +INSERT/UPDATE/DELETE timings by 10 milliseconds. In this case, the new index may not be worth +it. A new index is more valuable when SELECT timings are reduced and INSERT/UPDATE/DELETE +timings are unaffected. ## Finding Unused Indexes @@ -111,26 +102,32 @@ ORDER BY pg_relation_size(indexrelname::regclass) desc; ``` This query outputs a list containing all indexes that are never used and sorts -them by indexes sizes in descending order. This query can be useful to -determine if any previously indexes are useful after all. More information on +them by indexes sizes in descending order. This query helps in +determining whether existing indexes are still required. More information on the meaning of the various columns can be found at <https://www.postgresql.org/docs/current/monitoring-stats.html>. -Because the output of this query relies on the actual usage of your database it -may be affected by factors such as (but not limited to): +To determine if an index is still being used on production, use the following +Thanos query with your index name: + +```sql +sum(rate(pg_stat_user_indexes_idx_tup_read{env="gprd", indexrelname="index_ci_name", type="patroni-ci"}[5m])) +``` + +Because the query output relies on the actual usage of your database, it +may be affected by factors such as: - Certain queries never being executed, thus not being able to use certain indexes. - Certain tables having little data, resulting in PostgreSQL using sequence scans instead of index scans. -In other words, this data is only reliable for a frequently used database with -plenty of data and with as many GitLab features enabled (and being used) as -possible. +This data is only reliable for a frequently used database with +plenty of data, and using as many GitLab features as possible. ## Requirements for naming indexes -Indexes with complex definitions need to be explicitly named rather than +Indexes with complex definitions must be explicitly named rather than relying on the implicit naming behavior of migration methods. In short, that means you **must** provide an explicit name argument for an index created with one or more of the following options: @@ -144,16 +141,7 @@ created with one or more of the following options: ### Considerations for index names -Index names don't have any significance in the database, so they should -attempt to communicate intent to others. The most important rule to -remember is that generic names are more likely to conflict or be duplicated, -and should not be used. Some other points to consider: - -- For general indexes, use a template, like: `index_{table}_{column}_{options}`. -- For indexes added to solve a very specific problem, it may make sense - for the name to reflect their use. -- Identifiers in PostgreSQL have a maximum length of 63 bytes. -- Check `db/structure.sql` for conflicts and ideas. +Check our [Constraints naming conventions](database/constraint_naming_convention.md) page. ### Why explicit names are required @@ -172,7 +160,7 @@ end Creation of the second index would fail, because Rails would generate the same name for both indexes. -This is further complicated by the behavior of the `index_exists?` method. +This naming issue is further complicated by the behavior of the `index_exists?` method. It considers only the table name, column names, and uniqueness specification of the index when making a comparison. Consider: @@ -188,7 +176,7 @@ The call to `index_exists?` returns true if **any** index exists on `:my_table` and `:my_column`, and index creation is bypassed. The `add_concurrent_index` helper is a requirement for creating indexes -on populated tables. Since it cannot be used inside a transactional +on populated tables. Because it cannot be used inside a transactional migration, it has a built-in check that detects if the index already exists. In the event a match is found, index creation is skipped. Without an explicit name argument, Rails can return a false positive @@ -201,14 +189,14 @@ chance of error is greatly reduced. There may be times when an index is only needed temporarily. For example, in a migration, a column of a table might be conditionally -updated. To query which columns need to be updated within the -[query performance guidelines](query_performance.md), an index is needed that would otherwise -not be used. +updated. To query which columns must be updated in the +[query performance guidelines](query_performance.md), an index is needed +that would otherwise not be used. -In these cases, a temporary index should be considered. To specify a +In these cases, consider a temporary index. To specify a temporary index: -1. Prefix the index name with `tmp_` and follow the [naming conventions](database/constraint_naming_convention.md) and [requirements for naming indexes](#requirements-for-naming-indexes) for the rest of the name. +1. Prefix the index name with `tmp_` and follow the [naming conventions](database/constraint_naming_convention.md). 1. Create a follow-up issue to remove the index in the next (or future) milestone. 1. Add a comment in the migration mentioning the removal issue. @@ -237,10 +225,10 @@ on GitLab.com, the deployment process is blocked waiting for index creation to finish. To limit impact on GitLab.com, a process exists to create indexes -asynchronously during weekend hours. Due to generally lower levels of -traffic and lack of regular deployments, this process allows the -creation of indexes to proceed with a lower level of risk. The below -sections describe the steps required to use these features: +asynchronously during weekend hours. Due to generally lower traffic and fewer deployments, +index creation can proceed at a lower level of risk. + +### Schedule index creation for a low-impact time 1. [Schedule the index to be created](#schedule-the-index-to-be-created). 1. [Verify the MR was deployed and the index exists in production](#verify-the-mr-was-deployed-and-the-index-exists-in-production). @@ -291,12 +279,10 @@ migration as expected for other installations. The below block demonstrates how to create the second migration for the previous asynchronous example. -WARNING: -The responsibility lies on the individual writing the migrations to verify -the index exists in production before merging a second migration that -adds the index using `add_concurrent_index`. If the second migration is -deployed and the index has not yet been created, the index is created -synchronously when the second migration executes. +**WARNING:** +Verify that the index exists in production before merging a second migration with `add_concurrent_index`. +If the second migration is deployed before the index has been created, +the index is created synchronously when the second migration executes. ```ruby # in db/post_migrate/ diff --git a/doc/development/api_graphql_styleguide.md b/doc/development/api_graphql_styleguide.md index de6840b2c6c..37de7044765 100644 --- a/doc/development/api_graphql_styleguide.md +++ b/doc/development/api_graphql_styleguide.md @@ -475,17 +475,18 @@ end Developers can add [feature flags](../development/feature_flags/index.md) to GraphQL fields in the following ways: -- Add the `feature_flag` property to a field. This allows the field to be _hidden_ +- Add the [`feature_flag` property](#feature_flag-property) to a field. This allows the field to be _hidden_ from the GraphQL schema when the flag is disabled. -- Toggle the return value when resolving the field. +- [Toggle the return value](#toggle-the-value-of-a-field) when resolving the field. You can refer to these guidelines to decide which approach to use: - If your field is experimental, and its name or type is subject to - change, use the `feature_flag` property. + change, use the [`feature_flag` property](#feature_flag-property). - If your field is stable and its definition doesn't change, even after the flag is - removed, toggle the return value of the field instead. Note that + removed, [toggle the return value](#toggle-the-value-of-a-field) of the field instead. Note that [all fields should be nullable](#nullable-fields) anyway. +- If your field will be accessed from frontend using the `@include` or `@skip` directive, [do not use the `feature_flag` property](#frontend-and-backend-feature-flag-strategies). ### `feature_flag` property @@ -517,6 +518,20 @@ field :test_field, type: GraphQL::Types::String, feature_flag: :my_feature_flag ``` +### Frontend and Backend feature flag strategies + +#### Directives + +When feature flags are used in the frontend to control the `@include` and `@skip` directives, do not use the `feature_flag` +property on the server-side. For the accepted backend workaround, see [Toggle the value of a field](#toggle-the-value-of-a-field). It is recommended that the feature flag used in this approach be the same for frontend and backend. + +Even if the frontend directives evaluate to `@include:false` / `@skip:true`, the guarded entity is sent to the backend and matched +against the GraphQL schema. We would then get an exception due to a schema mismatch. See the [frontend GraphQL guide](../development/fe_guide/graphql.md#the-include-directive) for more guidance. + +#### Different versions of a query + +See the guide frontend GraphQL guide for [different versions of a query](../development/fe_guide/graphql.md#different-versions-of-a-query), and [why it is not the preferred approach](../development/fe_guide/graphql.md#avoiding-multiple-query-versions) + ### Toggle the value of a field This method of using feature flags for fields is to toggle the @@ -524,6 +539,12 @@ return value of the field. This can be done in the resolver, in the type, or even in a model method, depending on your preference and situation. +Consider also [marking the field as Alpha](#marking-schema-items-as-alpha) +while the value of the field can be toggled. You can +[change or remove Alpha fields at any time](#breaking-change-exemptions) without needing to deprecate them. +This also signals to consumers of the public GraphQL API that the field is not +meant to be used yet. + When applying a feature flag to toggle the value of a field, the `description` of the field must: @@ -537,6 +558,7 @@ Example: ```ruby field :foo, GraphQL::Types::String, null: true, + deprecated: { reason: :alpha, milestone: '10.0' }, description: 'Some test field. Returns `null`' \ 'if `my_feature_flag` feature flag is disabled.' @@ -2007,13 +2029,13 @@ end .to contain_exactly(a_graphql_entity_for(issue, :iid, :title, created_at: some_time)) ``` -- Use `GraphqlHelpers#empty_schema` to create an empty schema, rather than creating +- Use `GraphqlHelpers#empty_schema` to create an empty schema, rather than creating one by hand. For example: - + ```ruby # good let(:schema) { empty_schema } - + # bad let(:query_type) { GraphQL::ObjectType.new } let(:schema) { GraphQL::Schema.define(query: query_type, mutation: nil)} @@ -2024,7 +2046,7 @@ end ```ruby # good let(:query) { query_double(schema: GitlabSchema) } - + # bad let(:query) { double('Query', schema: GitlabSchema) } ``` @@ -2092,9 +2114,9 @@ end ```ruby type Types::IssueType.connection_type, null: true ``` - + However this might cause a cyclic definition, which can result in errors like: - + ```ruby NameError: uninitialized constant Resolvers::GroupIssuesResolver ``` @@ -2109,7 +2131,7 @@ end class IssueConnectionType < CountableConnectionType end end - + Types::IssueConnectionType.prepend_mod_with('Types::IssueConnectionType') ``` @@ -2120,22 +2142,22 @@ end ```ruby type "Types::IssueConnection", null: true ``` - + Only use this style if you are having spec failures. This is not intended to be a new pattern that we use. This issue may disappear after we've upgraded to `2.x`. -- There can be instances where a spec fails because the class is not loaded correctly. - It relates to the +- There can be instances where a spec fails because the class is not loaded correctly. + It relates to the [circular dependencies problem](https://github.com/rmosolgo/graphql-ruby/issues/1929) and [Adding field with resolver on a Type causes "Can't determine the return type " error on a different Type](https://github.com/rmosolgo/graphql-ruby/issues/3974). Unfortunately, the errors generated don't really indicate what the problem is. For example, - remove the quotes from the `Rspec.descrbe` in + remove the quotes from the `Rspec.descrbe` in [ee/spec/graphql/resolvers/compliance_management/merge_requests/compliance_violation_resolver_spec.rb](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/spec/graphql/resolvers/compliance_management/merge_requests/compliance_violation_resolver_spec.rb). Then run `rspec ee/spec/graphql/resolvers/compliance_management/merge_requests/compliance_violation_resolver_spec.rb`. - + This generates errors with the expectations. For example: - + ```ruby 1) Resolvers::ComplianceManagement::MergeRequests::ComplianceViolationResolver#resolve user is authorized filtering the results when given an array of project IDs finds the filtered compliance violations Failure/Error: expect(subject).to contain_exactly(compliance_violation) @@ -2145,10 +2167,10 @@ end the extra elements were: [#<MergeRequests::ComplianceViolation id: 5, violating_user_id: 27, merge_request_id: 5, reason: "approved_by_merge_request_author", severity_level: "high">] # ./ee/spec/graphql/resolvers/compliance_management/merge_requests/compliance_violation_resolver_spec.rb:55:in `block (6 levels) in <top (required)>' ``` - + However, this is not a case of the wrong result being generated, it's because of the loading order of the `ComplianceViolationResolver` class. - + The only way we've found to fix this is by quoting the class name in the spec. For example, changing ```ruby @@ -2198,7 +2220,7 @@ end [removed eventually](https://gitlab.com/gitlab-org/gitlab/-/issues/363121), and writing unit tests for resolvers/mutations is [already deprecated](#writing-unit-tests-deprecated) - + ## Notes about Query flow and GraphQL infrastructure The GitLab GraphQL infrastructure can be found in `lib/gitlab/graphql`. diff --git a/doc/development/application_limits.md b/doc/development/application_limits.md index 6c7213ab235..2826b8a3bc4 100644 --- a/doc/development/application_limits.md +++ b/doc/development/application_limits.md @@ -12,8 +12,7 @@ limits to GitLab. ## Documentation First of all, you have to gather information and decide which are the different -limits that are set for the different GitLab tiers. You also need to -coordinate with others to [document](../administration/instance_limits.md) +limits that are set for the different GitLab tiers. Coordinate with others to [document](../administration/instance_limits.md) and communicate those limits. There is a guide about [introducing application @@ -206,6 +205,6 @@ rate limiting architecture: 1. Making it possible to define and override limits per namespace / per plan. 1. Automatically generating documentation about what limits are implemented and what the defaults are. -1. Defining limits in a single place that is easy to find an explore. +1. Defining limits in a single place that can be found and explored. 1. Soft and hard limits, with support for notifying users when a limit is approaching. diff --git a/doc/development/application_slis/rails_request_apdex.md b/doc/development/application_slis/rails_request_apdex.md index f9613a14dd1..3e3cd100183 100644 --- a/doc/development/application_slis/rails_request_apdex.md +++ b/doc/development/application_slis/rails_request_apdex.md @@ -254,6 +254,6 @@ In the **Budget Attribution** row, the **Puma Apdex** log link shows you how many requests are not meeting a 1s or 5s target. Learn more about the content of the dashboard in the documentation for -[Dashboards for stage groups](../stage_group_dashboards.md). For more information +[Dashboards for stage groups](../stage_group_observability/index.md). For more information on our exploration of the error budget itself, read the infrastructure issue [Stage group error budget exploration dashboard](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1365). diff --git a/doc/development/appsec/index.md b/doc/development/appsec/index.md index 2ece3fdf4bf..8361170c3d2 100644 --- a/doc/development/appsec/index.md +++ b/doc/development/appsec/index.md @@ -1,32 +1,11 @@ --- -stage: Secure, Protect -group: all -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: index, dev, reference +redirect_to: '../index.md' +remove_date: '2022-09-23' --- -# Application Security development documentation +This document was moved to [another location](../index.md). -Development guides that are specific to the stages that work on Application Security features are listed here. - -Please go to [Application Security](../../user/application_security/index.md) if you are looking for documentation on how to use those features. - -## Namespaces - -Application Security code in the Rails monolith is organized into the following namespaces, which generally follows -the feature categories in the [Secure](https://about.gitlab.com/stages-devops-lifecycle/secure/) and [Protect](https://about.gitlab.com/stages-devops-lifecycle/protect/) stages. - -- `AppSec`: shared code. - - `AppSec::ContainerScanning`: Container Scanning code. - - `AppSec::Dast`: DAST code. - - `AppSec::DependencyScanning`: Dependency Scanning code. - - `AppSec::Fuzzing::API`: API Fuzzing code. - - `AppSec::Fuzzing::Coverage`: Coverage Fuzzing code. - - `AppSec::Fuzzing`: Shared fuzzing code. - - `AppSec::LicenseCompliance`: License Compliance code. - - `AppSec::Sast`: SAST code. - - `AppSec::SecretDetection`: Secret Detection code. - - `AppSec::VulnMgmt`: Vulnerability Management code. - -Most AppSec code does not conform to these namespace guidelines. When developing, make an effort -to move existing code into the appropriate namespace whenever possible. +<!-- This redirect file can be deleted after <2022-09-23>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/backend/ruby_style_guide.md b/doc/development/backend/ruby_style_guide.md index eff6ae7f217..c86f21d4bac 100644 --- a/doc/development/backend/ruby_style_guide.md +++ b/doc/development/backend/ruby_style_guide.md @@ -16,7 +16,7 @@ document the new rule. For every new guideline, add it in a new section and link [version history note](../documentation/versions.md#add-a-version-history-item) to provide context and serve as a reference. -Just because something is listed here does not mean it cannot be reopened for discussion. +Everything listed here can be [reopened for discussion](https://about.gitlab.com/handbook/values/#disagree-commit-and-disagree). ## Instance variable access using `attr_reader` diff --git a/doc/development/cicd/pipeline_wizard.md b/doc/development/cicd/pipeline_wizard.md index 608c21778c0..7a0b70bd8e8 100644 --- a/doc/development/cicd/pipeline_wizard.md +++ b/doc/development/cicd/pipeline_wizard.md @@ -227,3 +227,21 @@ Use as `widget: list`. This inserts a `list` in the YAML file. | `invalidFeedback` | **{dotted-circle}** No | string | Help text displayed when the pattern validation fails. | | `default` | **{dotted-circle}** No | list | The default value for the list | | `id` | **{dotted-circle}** No | string | The input field ID is usually autogenerated but can be overridden by providing this property. | + +#### Checklist + +Use as `widget: checklist`. This inserts a list of checkboxes that need to +be checked before proceeding to the next step. + +| Name | Required | Type | Description | +|---------|------------------------|--------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| `title` | **{dotted-circle}** No | string | A title above the checklist items. | +| `items` | **{dotted-circle}** No | list | A list of items that need to be checked. Each item corresponds to one checkbox, and can be a string or [checklist item](#checklist-item). | + +##### Checklist Item + +| Name | Required | Type | Description | +|--------|------------------------|---------|-----------------------------------------| +| `text` | **{check-circle}** Yes | string | A title above the checklist items. | +| `help` | **{dotted-circle}** No | string | Help text explaining the item. | +| `id` | **{dotted-circle}** No | string | The input field ID is usually autogenerated but can be overridden by providing this property. | diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index 8d88e7155a2..4ea7a9d960c 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -289,9 +289,32 @@ Please read [versioning](#versioning) section for introducing breaking change sa ## Versioning -Versioning allows you to introduce a new template without modifying the existing -one. This process is useful when we need to introduce a breaking change, -but don't want to affect the existing projects that depends on the current template. +To introduce a breaking change without affecting the existing projects that depend on +the current template, use [stable](#stable-version) and [latest](#latest-version) versioning. + +Stable templates usually only receive breaking changes in major version releases, while +latest templates can receive breaking changes in any release. In major release milestones, +the latest template is made the new stable template (and the latest template might be deleted). + +Adding a latest template is safe, but comes with a maintenance burden: + +- GitLab has to choose a DRI to overwrite the stable template with the contents of the + latest template at the next major release of GitLab. The DRI is responsible for + supporting users who have trouble with the change. +- When we make a new non-breaking change, both the stable and latest templates must be updated + to match, as must as possible. +- A latest template could remain for longer than planned because many users could + directly depend on it continuing to exist. + +Before adding a new latest template, see if the change can be made to the stable +template instead, even if it's a breaking change. If the template is intended for copy-paste +usage only, it might be possible to directly change the stable version. Before changing +the stable template with a breaking change in a minor milestone, make sure: + +- It's a [pipeline template](#template-types) and it has a [code comment](#explain-requirements-and-expectations) + explaining that it's not designed to be used with the `includes`. +- The [CI/CD template usage metrics](#add-metrics) doesn't show any usage. If the metrics + show zero usage for the template, the template is not actively being used with `include`. ### Stable version @@ -393,7 +416,9 @@ is updated in a major version GitLab release. ### Add metrics -Every CI/CD template must also have metrics defined to track their use. +Every CI/CD template must also have metrics defined to track their use. The CI/CD template monthly usage report +can be found in [Sisense (GitLab team members only)](https://app.periscopedata.com/app/gitlab/785953/Pipeline-Authoring-Dashboard?widget=13440051&udv=0). +Double click a template to see the graph for that single template. To add a metric definition for a new template: diff --git a/doc/development/code_review.md b/doc/development/code_review.md index a6976271ddf..1225260e600 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -71,26 +71,34 @@ It picks reviewers and maintainers from the list at the [engineering projects](https://about.gitlab.com/handbook/engineering/projects/) page, with these behaviors: -1. It doesn't pick people whose Slack or [GitLab status](../user/profile/index.md#set-your-current-status): - - Contains the string `OOO`, `PTO`, `Parental Leave`, or `Friends and Family`. - - GitLab user **Busy** indicator is set to `True`. - - Emoji is from one of these categories: - - **On leave** - 🌴 `:palm_tree:`, 🏖️ `:beach:`, ⛱ `:beach_umbrella:`, 🏖 `:beach_with_umbrella:`, 🌞 `:sun_with_face:`, 🎡 `:ferris_wheel:` - - **Out sick** - 🌡️ `:thermometer:`, 🤒 `:face_with_thermometer:` - - **At capacity** - 🔴 `:red_circle:` - - **Focus mode** - 💡 `:bulb:` (focusing on their team's work) -1. [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) - are three times as likely to be picked as other reviewers. -1. Team members whose Slack or [GitLab status](../user/profile/index.md#set-your-current-status) emoji - is 🔵 `:large_blue_circle:` are more likely to be picked. This applies to both reviewers and trainee maintainers. - - Reviewers with 🔵 `:large_blue_circle:` are two times as likely to be picked as other reviewers. - - Trainee maintainers with 🔵 `:large_blue_circle:` are four times as likely to be picked as other reviewers. -1. People whose [GitLab status](../user/profile/index.md#set-your-current-status) emoji - is 🔶 `:large_orange_diamond:` or 🔸 `:small_orange_diamond:` are half as likely to be picked. -1. It always picks the same reviewers and maintainers for the same - branch name (unless their out-of-office (`OOO`) status changes, as in point 1). It - removes leading `ce-` and `ee-`, and trailing `-ce` and `-ee`, so - that it can be stable for backport branches. +- It doesn't pick people whose Slack or [GitLab status](../user/profile/index.md#set-your-current-status): + - Contains the string `OOO`, `PTO`, `Parental Leave`, or `Friends and Family`. + - GitLab user **Busy** indicator is set to `True`. + - Emoji is from one of these categories: + - **On leave** - 🌴 `:palm_tree:`, 🏖️ `:beach:`, ⛱ `:beach_umbrella:`, 🏖 `:beach_with_umbrella:`, 🌞 `:sun_with_face:`, 🎡 `:ferris_wheel:` + - **Out sick** - 🌡️ `:thermometer:`, 🤒 `:face_with_thermometer:` + - **At capacity** - 🔴 `:red_circle:` + - **Focus mode** - 💡 `:bulb:` (focusing on their team's work) +- It doesn't pick people who are already assigned a number of reviews that is equal to + or greater than their chosen "review limit". The review limit is the maximum number of + reviews people are ready to handle at a time. Set a review limit by using one of the following + as a Slack or [GitLab status](../user/profile/index.md#set-your-current-status): + - 0️⃣ - `:zero:` (similar to `:red_circle:`) + - 1️⃣ - `:one:` + - 2️⃣ - `:two:` + - 3️⃣ - `:three:` + - 4️⃣ - `:four:` + - 5️⃣ - `:five:` +- Team members whose Slack or [GitLab status](../user/profile/index.md#set-your-current-status) emoji + is 🔵 `:large_blue_circle:` are more likely to be picked. This applies to both reviewers and trainee maintainers. + - Reviewers with 🔵 `:large_blue_circle:` are two times as likely to be picked as other reviewers. + - [Trainee maintainers](https://about.gitlab.com/handbook/engineering/workflow/code-review/#trainee-maintainer) with 🔵 `:large_blue_circle:` are three times as likely to be picked as other reviewers. +- People whose [GitLab status](../user/profile/index.md#set-your-current-status) emoji + is 🔶 `:large_orange_diamond:` or 🔸 `:small_orange_diamond:` are half as likely to be picked. +- It always picks the same reviewers and maintainers for the same + branch name (unless their out-of-office (`OOO`) status changes, as in point 1). It + removes leading `ce-` and `ee-`, and trailing `-ce` and `-ee`, so + that it can be stable for backport branches. The [Roulette dashboard](https://gitlab-org.gitlab.io/gitlab-roulette) contains: @@ -131,7 +139,7 @@ with [domain expertise](#domain-experts). 1. If your merge request includes documentation changes, it must be **approved by a [Technical writer](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments)**, based on assignments in the appropriate [DevOps stage group](https://about.gitlab.com/handbook/product/categories/#devops-stages). -1. If your merge request includes changes to development guidelines, follow the [review process](index.md#development-guidelines-review) and get the approvals accordingly. +1. If your merge request includes changes to development guidelines, follow the [review process](development_processes.md#development-guidelines-review) and get the approvals accordingly. 1. If your merge request includes end-to-end **and** non-end-to-end changes (*4*), it must be **approved by a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors)**. 1. If your merge request only includes end-to-end changes (*4*) **or** if the MR author is a [Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), it must be **approved by a [Quality maintainer](https://about.gitlab.com/handbook/engineering/projects/#gitlab_maintainers_qa)** @@ -275,7 +283,7 @@ This saves reviewers time and helps authors catch mistakes earlier. Verify that the merge request meets all [contribution acceptance criteria](contributing/merge_request_workflow.md#contribution-acceptance-criteria). If a merge request is too large, fixes more than one issue, or implements more -than one feature, you should guide the author towards spltting the merge request +than one feature, you should guide the author towards splitting the merge request into smaller merge requests. When you are confident @@ -300,11 +308,18 @@ Because a maintainer's job only depends on their knowledge of the overall GitLab codebase, and not that of any specific domain, they can review, approve, and merge merge requests from any team and in any product area. -If a merge request is too large, fixes more than one issue, or implements more -than one feature, the maintainer can ask the author to make the merge request -smaller. Request the previous reviewer, or a merge request coach to help guide -the author on how to split the merge request, and to review the resulting -changes. +A maintainer should ask the author to make a merge request smaller if it is: + +- Too large. +- Fixes more than one issue. +- Implements more than one feature. +- Has a high complexity resulting in additional risk. + +The maintainer, any of the +reviewers, or a merge request coach can step up to help the author to divide work +into smaller iterations, and guide the author on how to split the merge request. +The author may choose to request that the current maintainers and reviewers review the split MRs +or request a new group of maintainers and reviewers. Maintainers do their best to also review the specifics of the chosen solution before merging, but as they are not necessarily [domain experts](#domain-experts), they may be poorly diff --git a/doc/development/contributing/index.md b/doc/development/contributing/index.md index 182d00d52ab..12fd7c3dc12 100644 --- a/doc/development/contributing/index.md +++ b/doc/development/contributing/index.md @@ -240,4 +240,4 @@ For information on how to contribute documentation, see GitLab ## Getting an Enterprise Edition License If you need a license for contributing to an EE-feature, see -[relevant information](https://about.gitlab.com/handbook/marketing/community-relations/code-contributor-program/#for-contributors-to-the-gitlab-enterprise-edition-ee). +[relevant information](https://about.gitlab.com/handbook/marketing/community-relations/code-contributor-program/#contributing-to-the-gitlab-enterprise-edition-ee). diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index 97c8c179e09..c6d977cf5ad 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -68,7 +68,7 @@ labels, you can _always_ add the type, stage, group, and often the category/feat Type labels are very important. They define what kind of issue this is. Every issue should have one and only one. -The current type labels are [available in the handbook](https://about.gitlab.com/handbook/engineering/metrics/#work-type-classification) +The SSOT for type and subtype labels is [available in the handbook](https://about.gitlab.com/handbook/engineering/metrics/#work-type-classification). A number of type labels have a priority assigned to them, which automatically makes them float to the top, depending on their importance. diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index 003df4fe078..d2b231ebc7c 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -58,7 +58,7 @@ itself, increasing visibility. ## Development guidelines -Danger code is Ruby code, so all our [usual backend guidelines](index.md#backend-guides) +Danger code is Ruby code, so all our [usual backend guidelines](feature_development.md#backend-guides) continue to apply. However, there are a few things that deserve special emphasis. ### When to use Danger @@ -175,15 +175,7 @@ at GitLab so far: - Database review - Documentation review - Merge request metrics -- Reviewer roulette. Reviewers and maintainers are chosen based on: - - Their roles (backend, frontend, database, etc). - - Their availability: - - No "OOO"/"PTO"/"Parental Leave" in their GitLab or Slack status. - - No `:red_circle:`/`:palm_tree:`/`:beach:`/`:beach_umbrella:`/`:beach_with_umbrella:` emojis in GitLab or Slack status. - - (Experimental) Their time zone: people for which the local hour is between - 6 AM and 2 PM are eligible to be picked. This is to ensure they have a good - chance to get to perform a review during their current work day. The experimentation is tracked in - [this issue](https://gitlab.com/gitlab-org/quality/team-tasks/-/issues/563) +- [Reviewer roulette](code_review.md#reviewer-roulette) - Single codebase effort ## Limitations diff --git a/doc/development/database/add_foreign_key_to_existing_column.md b/doc/development/database/add_foreign_key_to_existing_column.md index 9842814816f..7a18da2223f 100644 --- a/doc/development/database/add_foreign_key_to_existing_column.md +++ b/doc/development/database/add_foreign_key_to_existing_column.md @@ -64,18 +64,14 @@ emails = Email.where(user_id: 1) # returns emails for the deleted user Add a `NOT VALID` foreign key constraint to the table, which enforces consistency on the record changes. -[Using the `with_lock_retries` helper method is advised when performing operations on high-traffic tables](../migration_style_guide.md#when-to-use-the-helper-method), -in this case, if the table or the foreign table is a high-traffic table, we should use the helper method. - In the example above, you'd be still able to update records in the `emails` table. However, when you'd try to update the `user_id` with non-existent value, the constraint causes a database error. Migration file for adding `NOT VALID` foreign key: ```ruby -class AddNotValidForeignKeyToEmailsUser < Gitlab::Database::Migration[1.0] +class AddNotValidForeignKeyToEmailsUser < Gitlab::Database::Migration[2.0] def up - # safe to use: it requires short lock on the table since we don't validate the foreign key - add_foreign_key :emails, :users, on_delete: :cascade, validate: false + add_concurrent_foreign_key :emails, :users, on_delete: :cascade, validate: false end def down @@ -84,8 +80,14 @@ class AddNotValidForeignKeyToEmailsUser < Gitlab::Database::Migration[1.0] end ``` +Adding a foreign key without validating it is a fast operation. It only requires a +short lock on the table before being able to enforce the constraint on new data. +We do still want to enable lock retries for high traffic and large tables. +`add_concurrent_foreign_key` does this for us, and also checks if the foreign key already exists. + WARNING: -Avoid using the `add_foreign_key` constraint more than once per migration file, unless the source and target tables are identical. +Avoid using `add_foreign_key` or `add_concurrent_foreign_key` constraints more than +once per migration file, unless the source and target tables are identical. #### Data migration to fix existing records @@ -98,7 +100,7 @@ In case the data volume is higher (>1000 records), it's better to create a backg Example for cleaning up records in the `emails` table in a database migration: ```ruby -class RemoveRecordsWithoutUserFromEmailsTable < Gitlab::Database::Migration[1.0] +class RemoveRecordsWithoutUserFromEmailsTable < Gitlab::Database::Migration[2.0] disable_ddl_transaction! class Email < ActiveRecord::Base @@ -121,6 +123,7 @@ end ### Validate the foreign key Validating the foreign key scans the whole table and makes sure that each relation is correct. +Fortunately, this does not lock the source table (`users`) while running. NOTE: When using [background migrations](background_migrations.md), foreign key validation should happen in the next GitLab release. @@ -130,7 +133,7 @@ Migration file for validating the foreign key: ```ruby # frozen_string_literal: true -class ValidateForeignKeyOnEmailUsers < Gitlab::Database::Migration[1.0] +class ValidateForeignKeyOnEmailUsers < Gitlab::Database::Migration[2.0] def up validate_foreign_key :emails, :user_id end diff --git a/doc/development/database/batched_background_migrations.md b/doc/development/database/batched_background_migrations.md index 6d3d5fa7f92..f3ea82b5c61 100644 --- a/doc/development/database/batched_background_migrations.md +++ b/doc/development/database/batched_background_migrations.md @@ -244,8 +244,6 @@ background migration. ```ruby class QueueBackfillRoutesNamespaceId < Gitlab::Database::Migration[2.0] - disable_ddl_transaction! - MIGRATION = 'BackfillRouteNamespaceId' DELAY_INTERVAL = 2.minutes @@ -316,6 +314,137 @@ background migration. After the batched migration is completed, you can safely depend on the data in `routes.namespace_id` being populated. +### Batching over non-distinct columns + +The default batching strategy provides an efficient way to iterate over primary key columns. +However, if you need to iterate over columns where values are not unique, you must use a +different batching strategy. + +The `LooseIndexScanBatchingStrategy` batching strategy uses a special version of [`EachBatch`](../iterating_tables_in_batches.md#loose-index-scan-with-distinct_each_batch) +to provide efficient and stable iteration over the distinct column values. + +This example shows a batched background migration where the `issues.project_id` column is used as +the batching column. + +Database post-migration: + +```ruby +class ProjectsWithIssuesMigration < Gitlab::Database::Migration[2.0] + MIGRATION = 'BatchProjectsWithIssues' + INTERVAL = 2.minutes + BATCH_SIZE = 5000 + SUB_BATCH_SIZE = 500 + restrict_gitlab_migration gitlab_schema: :gitlab_main + + disable_ddl_transaction! + def up + queue_batched_background_migration( + MIGRATION, + :issues, + :project_id, + job_interval: INTERVAL, + batch_size: BATCH_SIZE, + batch_class_name: 'LooseIndexScanBatchingStrategy', # Override the default batching strategy + sub_batch_size: SUB_BATCH_SIZE + ) + end + + def down + delete_batched_background_migration(MIGRATION, :issues, :project_id, []) + end +end +``` + +Implementing the background migration class: + +```ruby +module Gitlab + module BackgroundMigration + class BatchProjectsWithIssues < Gitlab::BackgroundMigration::BatchedMigrationJob + include Gitlab::Database::DynamicModelHelpers + + def perform + distinct_each_batch(operation_name: :backfill_issues) do |batch| + project_ids = batch.pluck(batch_column) + # do something with the distinct project_ids + end + end + end + end +end +``` + +### Adding filters to the initial batching + +By default, when creating background jobs to perform the migration, batched background migrations will iterate over the full specified table. This is done using the [`PrimaryKeyBatchingStrategy`](https://gitlab.com/gitlab-org/gitlab/-/blob/c9dabd1f4b8058eece6d8cb4af95e9560da9a2ee/lib/gitlab/database/migrations/batched_background_migration_helpers.rb#L17). This means if there are 1000 records in the table and the batch size is 100, there will be 10 jobs. For illustrative purposes, `EachBatch` is used like this: + +```ruby +# PrimaryKeyBatchingStrategy +Projects.all.each_batch(of: 100) do |relation| + relation.where(foo: nil).update_all(foo: 'bar') # this happens in each background job +end +``` + +There are cases where we only need to look at a subset of records. Perhaps we only need to update 1 out of every 10 of those 1000 records. It would be best if we could apply a filter to the initial relation when the jobs are created: + +```ruby +Projects.where(foo: nil).each_batch(of: 100) do |relation| + relation.update_all(foo: 'bar') +end +``` + +In the `PrimaryKeyBatchingStrategy` example, we do not know how many records will be updated in each batch. In the filtered example, we know exactly 100 will be updated with each batch. + +The `PrimaryKeyBatchingStrategy` contains [a method that can be overwritten](https://gitlab.com/gitlab-org/gitlab/-/blob/dd1e70d3676891025534dc4a1e89ca9383178fe7/lib/gitlab/background_migration/batching_strategies/primary_key_batching_strategy.rb#L38-52) to apply additional filtering on the initial `EachBatch`. + +We can accomplish this by: + +1. Create a new class that inherits from `PrimaryKeyBatchingStrategy` and overrides the method using the desired filter (this may be the same filter used in the sub-batch): + + ```ruby + # frozen_string_literal: true + + module GitLab + module BackgroundMigration + module BatchingStrategies + class FooStrategy < PrimaryKeyBatchingStrategy + def apply_additional_filters(relation, job_arguments: [], job_class: nil) + relation.where(foo: nil) + end + end + end + end + end + ``` + +1. In the post-deployment migration that queues the batched background migration, specify the new batching strategy using the `batch_class_name` parameter: + + ```ruby + class BackfillProjectsFoo < Gitlab::Database::Migration[2.0] + MIGRATION = 'BackfillProjectsFoo' + DELAY_INTERVAL = 2.minutes + BATCH_CLASS_NAME = 'FooStrategy' + + restrict_gitlab_migration gitlab_schema: :gitlab_main + + def up + queue_batched_background_migration( + MIGRATION, + :routes, + :id, + job_interval: DELAY_INTERVAL, + batch_class_name: BATCH_CLASS_NAME + ) + end + + def down + delete_batched_background_migration(MIGRATION, :routes, :id, []) + end + end + ``` + +When applying a batching strategy, it is important to ensure the filter properly covered by an index to optimize `EachBatch` performance. See [the `EachBatch` docs for more information](../iterating_tables_in_batches.md). + ## Testing Writing tests is required for: @@ -357,7 +486,7 @@ You can view failures in two ways: - Via GitLab logs: 1. After running a batched background migration, if any jobs fail, - view the logs in [Kibana](https://log.gprd.gitlab.net/goto/5f06a57f768c6025e1c65aefb4075694). + view the logs in [Kibana](https://log.gprd.gitlab.net/goto/4cb43f40-f861-11ec-b86b-d963a1a6788e). View the production Sidekiq log and filter for: - `json.new_state: failed` diff --git a/doc/development/database/constraint_naming_convention.md b/doc/development/database/constraint_naming_convention.md index 72f16c20559..2f0b8bf0463 100644 --- a/doc/development/database/constraint_naming_convention.md +++ b/doc/development/database/constraint_naming_convention.md @@ -25,5 +25,7 @@ The intent is not to retroactively change names in existing databases but rather ## Observations +- Check `db/structure.sql` for conflicts. - Prefixes are preferred over suffices because they make it easier to identify the type of a given constraint quickly, as well as group them alphabetically; - The `_and_` that joins column names can be omitted to keep the identifiers under the 63 characters' length limit defined by PostgreSQL. Additionally, the notation may be abbreviated to the best of our ability if struggling to keep under this limit. +- For indexes added to solve a very specific problem, it may make sense for the name to reflect their use. diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md index dec51d484fd..6aa1b9c40ff 100644 --- a/doc/development/database/loose_foreign_keys.md +++ b/doc/development/database/loose_foreign_keys.md @@ -66,8 +66,6 @@ The tool ensures that all aspects of swapping a foreign key are covered. This in - Updating `db/structure.sql` with the new migration. - Updating `lib/gitlab/database/gitlab_loose_foreign_keys.yml` to add the new loose foreign key. - Creating or updating a model's specs to ensure that the loose foreign key is properly supported. -- Creating a new branch, commit, push, and creating a merge request on GitLab.com. -- Creating a merge request template with all the necessary details to validate the safety of the foreign key removal. The tool is located at `scripts/decomposition/generate-loose-foreign-key`: @@ -77,9 +75,7 @@ $ scripts/decomposition/generate-loose-foreign-key -h Usage: scripts/decomposition/generate-loose-foreign-key [options] <filters...> -c, --cross-schema Show only cross-schema foreign keys -n, --dry-run Do not execute any commands (dry run) - -b, --[no-]branch Create or not a new branch -r, --[no-]rspec Create or not a rspecs automatically - -m, --milestone MILESTONE Specify custom milestone (current: 14.8) -h, --help Prints this help ``` diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index 7badd7f76fa..9641ea37002 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -6,8 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Multiple Databases -To scale GitLab, the we are -[decomposing the GitLab application database into multiple databases](https://gitlab.com/groups/gitlab-org/-/epics/6168). +To allow GitLab to scale further we +[decomposed the GitLab application database into multiple +databases](https://gitlab.com/groups/gitlab-org/-/epics/6168). The two databases +are `main` and `ci`. GitLab supports being run with either one database or two databases. +On GitLab.com we are using two separate databases. ## GitLab Schema @@ -23,6 +26,7 @@ Each table of GitLab needs to have a `gitlab_schema` assigned: - `gitlab_main`: describes all tables that are being stored in the `main:` database (for example, like `projects`, `users`). - `gitlab_ci`: describes all CI tables that are being stored in the `ci:` database (for example, `ci_pipelines`, `ci_builds`). +- `gitlab_geo`: describes all Geo tables that are being stored in the `geo:` database (for example, like `project_registry`, `secondary_usage_data`). - `gitlab_shared`: describe all application tables that contain data across all decomposed databases (for example, `loose_foreign_keys_deleted_records`) for models that inherit from `Gitlab::Database::SharedModel`. - `gitlab_internal`: describe all internal tables of Rails and PostgreSQL (for example, `ar_internal_metadata`, `schema_migrations`, `pg_*`). - `...`: more schemas to be introduced with additional decomposed databases @@ -31,6 +35,7 @@ The usage of schema enforces the base class to be used: - `ApplicationRecord` for `gitlab_main` - `Ci::ApplicationRecord` for `gitlab_ci` +- `Geo::TrackingBase` for `gitlab_geo` - `Gitlab::Database::SharedModel` for `gitlab_shared` ### The impact of `gitlab_schema` @@ -40,7 +45,7 @@ The `gitlab_schema` primary purpose is to introduce a barrier between different This is used as a primary source of classification for: -- [Discovering cross-joins across tables from different schemas](#removing-joins-between-ci_-and-non-ci_-tables) +- [Discovering cross-joins across tables from different schemas](#removing-joins-between-ci-and-non-ci-tables) - [Discovering cross-database transactions across tables from different schemas](#removing-cross-database-transactions) ### The special purpose of `gitlab_shared` @@ -72,10 +77,6 @@ Read [Migrations for Multiple Databases](migrations_for_multiple_databases.md). ## CI/CD Database -> Support for configuring the GitLab Rails application to use a distinct -database for CI/CD tables was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64289) -in GitLab 14.1. This feature is still under development, and is not ready for production use. - ### Configure single database By default, GDK is configured to run with multiple databases. @@ -107,32 +108,14 @@ NOTE: The `validate_cross_joins!` method in `spec/support/database/prevent_cross the corresponding documentation URL used in `spec/support/database/prevent_cross_joins.rb`. --> -### Removing joins between `ci_*` and non `ci_*` tables +### Removing joins between `ci` and non `ci` tables Queries that join across databases raise an error. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68620) in GitLab 14.3, for new queries only. Pre-existing queries do not raise an error. -We are planning on moving all the `ci_*` tables to a separate database, so -referencing `ci_*` tables with other tables will not be possible. This means, -that using any kind of `JOIN` in SQL queries will not work. We have identified -already many such examples that need to be fixed in -<https://gitlab.com/groups/gitlab-org/-/epics/6289> . - -#### Path to removing cross-database joins - -The following steps are the process to remove cross-database joins between -`ci_*` and non `ci_*` tables: - -1. **{check-circle}** Add all failing specs to the [`cross-join-allowlist.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/support/database/cross-join-allowlist.yml) - file. -1. **{check-circle}** Find the code that caused the spec failure and wrap the isolated code - in [`allow_cross_joins_across_databases`](#allowlist-for-existing-cross-joins). - Link to a new issue assigned to the correct team to remove the specs from the - `cross-join-allowlist.yml` file. -1. **{dotted-circle}** Remove the `cross-join-allowlist.yml` file and stop allowing - whole test files. -1. **{dotted-circle}** Fix the problem and remove the `allow_cross_joins_across_databases` call. -1. **{dotted-circle}** Fix all the cross-joins and remove the `allow_cross_joins_across_databases` method. +Because GitLab can be run with multiple separate databases, referencing `ci` +tables with non `ci` tables in a single query is not possible. Therefore, +using any kind of `JOIN` in SQL queries will not work. #### Suggestions for removing cross-database joins @@ -416,13 +399,10 @@ query or look at other patterns described above for removing cross-joins. #### How to validate you have correctly removed a cross-join -Using RSpec tests, you can validate all SQL queries within a code block to -ensure that none of them are joining across the two databases. This is a useful -tool to confirm you have correctly fixed an existing cross-join. - -At some point in the future we will have fixed all cross-joins and this tool -will run by default in all tests. For now, the tool needs to be explicitly enabled -for your test. +RSpec is configured to automatically validate all SQL queries do not join +across databases. If this validation is disabled in +`spec/support/database/cross-join-allowlist.yml` then you can still validate an +isolated code block using `with_cross_joins_prevented`. You can use this method like so: @@ -553,12 +533,11 @@ The `MyAsyncConsistencyJob` would also attempt to update the timestamp if they d At this point, we don't have the tooling (we might not even need it) to ensure similar consistency characteristics as we had with one database. If you think that the code you're working on requires -these properties, then you can disable the cross-database modification check by wrapping to -offending database queries with a block and create a follow-up issue mentioning the sharding group -(`gitlab-org/sharding-group`). +these properties, then you can disable the cross-database modification check in your tests by wrapping the +offending test code with a block and create a follow-up issue. ```ruby -Gitlab::Database.allow_cross_joins_across_databases(url: 'gitlab issue URL') do +allow_cross_database_modification_within_transaction(url: 'gitlab issue URL') do ApplicationRecord.transaction do ci_build.update!(updated_at: Time.current) # UPDATE on CI DB ci_build.project.update!(updated_at: Time.current) # UPDATE on Main DB @@ -567,7 +546,7 @@ end ``` Don't hesitate to reach out to the -[sharding group](https://about.gitlab.com/handbook/engineering/development/enablement/sharding/) +[pods group](https://about.gitlab.com/handbook/engineering/development/enablement/data_stores/pods/) for advice. ##### Avoid `dependent: :nullify` and `dependent: :destroy` across databases diff --git a/doc/development/database/transaction_guidelines.md b/doc/development/database/transaction_guidelines.md index d96d11f05a5..255de19a420 100644 --- a/doc/development/database/transaction_guidelines.md +++ b/doc/development/database/transaction_guidelines.md @@ -132,12 +132,12 @@ end build_1 = Ci::Build.find(1) build_2 = Ci::Build.find(2) -ActiveRecord::Base.transaction do +ApplicationRecord.transaction do build_1.touch build_2.touch end ``` -The `ActiveRecord::Base` class uses a different database connection than the `Ci::Build` records. +The `ApplicationRecord` class uses a different database connection than the `Ci::Build` records. The two statements in the transaction block are not part of the transaction and are rolled back in case something goes wrong. They act as 3rd part calls. diff --git a/doc/development/deprecation_guidelines/img/deprecation_removal_process.png b/doc/development/deprecation_guidelines/img/deprecation_removal_process.png Binary files differindex 99642ebbae0..594e15861b0 100644 --- a/doc/development/deprecation_guidelines/img/deprecation_removal_process.png +++ b/doc/development/deprecation_guidelines/img/deprecation_removal_process.png diff --git a/doc/development/deprecation_guidelines/index.md b/doc/development/deprecation_guidelines/index.md index 7fbe2261f4d..4e1d2e22e78 100644 --- a/doc/development/deprecation_guidelines/index.md +++ b/doc/development/deprecation_guidelines/index.md @@ -41,7 +41,9 @@ changes](../contributing/index.md#breaking-changes) to GitLab features. Deprecations should be announced on the [Deprecated feature removal schedule](../../update/deprecations.md). -For steps to create a deprecation entry, see [Deprecations](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations). +Do not include the deprecation announcement in the merge request that introduces a code change for the deprecation. +Use a separate MR to create a deprecation entry. For steps to create a deprecation entry, see +[Deprecations](https://about.gitlab.com/handbook/marketing/blog/release-posts/#deprecations). ## When can a feature be removed/changed? diff --git a/doc/development/development_processes.md b/doc/development/development_processes.md new file mode 100644 index 00000000000..e199aedd3f5 --- /dev/null +++ b/doc/development/development_processes.md @@ -0,0 +1,124 @@ +--- +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + +# Development processes + +Consult these topics for information on development processes for contributing to GitLab. + +## Processes + +Must-reads: + +- [Guide on adapting existing and introducing new components](architecture.md#adapting-existing-and-introducing-new-components) +- [Code review guidelines](code_review.md) for reviewing code and having code + reviewed +- [Database review guidelines](database_review.md) for reviewing + database-related changes and complex SQL queries, and having them reviewed +- [Secure coding guidelines](secure_coding_guidelines.md) +- [Pipelines for the GitLab project](pipelines.md) + +Complementary reads: + +- [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md) +- [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer) +- [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers) +- [Guidelines for implementing Enterprise Edition features](ee_features.md) +- [Adding a new service component to GitLab](adding_service_component.md) +- [Guidelines for changelogs](changelog.md) +- [Dependencies](dependencies.md) +- [Danger bot](dangerbot.md) +- [Requesting access to ChatOps on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLab team members) + +### Development guidelines review + +When you submit a change to the GitLab development guidelines, who +you ask for reviews depends on the level of change. + +#### Wording, style, or link changes + +Not all changes require extensive review. For example, MRs that don't change the +content's meaning or function can be reviewed, approved, and merged by any +maintainer or Technical Writer. These can include: + +- Typo fixes. +- Clarifying links, such as to external programming language documentation. +- Changes to comply with the [Documentation Style Guide](documentation/index.md) + that don't change the intent of the documentation page. + +#### Specific changes + +If the MR proposes changes that are limited to a particular stage, group, or team, +request a review and approval from an experienced GitLab Team Member in that +group. For example, if you're documenting a new internal API used exclusively by +a given group, request an engineering review from one of the group's members. + +After the engineering review is complete, assign the MR to the +[Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) +in the modified documentation page's metadata. +If the page is not assigned to a specific group, follow the +[Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). + +#### Broader changes + +Some changes affect more than one group. For example: + +- Changes to [code review guidelines](code_review.md). +- Changes to [commit message guidelines](contributing/merge_request_workflow.md#commit-messages-guidelines). +- Changes to guidelines in [feature flags in development of GitLab](feature_flags/). +- Changes to [feature flags documentation guidelines](documentation/feature_flags.md). + +In these cases, use the following workflow: + +1. Request a peer review from a member of your team. +1. Request a review and approval of an Engineering Manager (EM) + or Staff Engineer who's responsible for the area in question: + + - [Frontend](https://about.gitlab.com/handbook/engineering/frontend/) + - [Backend](https://about.gitlab.com/handbook/engineering/) + - [Database](https://about.gitlab.com/handbook/engineering/development/database/) + - [User Experience (UX)](https://about.gitlab.com/handbook/engineering/ux/) + - [Security](https://about.gitlab.com/handbook/engineering/security/) + - [Quality](https://about.gitlab.com/handbook/engineering/quality/) + - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/) + - [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/) + - [Technical Writing](https://about.gitlab.com/handbook/engineering/ux/technical-writing/) + + You can skip this step for MRs authored by EMs or Staff Engineers responsible + for their area. + + If there are several affected groups, you may need approvals at the + EM/Staff Engineer level from each affected area. + +1. After completing the reviews, consult with the EM/Staff Engineer + author / approver of the MR. + + If this is a significant change across multiple areas, request final review + and approval from the VP of Development, the DRI for Development Guidelines, + @clefelhocz1. + +1. After all approvals are complete, assign the MR to the + [Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) + in the modified documentation page's metadata. + If the page is not assigned to a specific group, follow the + [Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). + The Technical Writer may ask for additional approvals as previously suggested before merging the MR. + +### Reviewer values + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57293) in GitLab 14.1. + +As a reviewer or as a reviewee, make sure to familiarize yourself with +the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/reviewer-values/) we strive for at GitLab. + +## Language-specific guides + +### Go guides + +- [Go Guidelines](go_guide/index.md) + +### Shell Scripting guides + +- [Shell scripting standards and style guidelines](shell_scripting_guide/index.md) diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index 1f270a2b5ee..92c34c01e5d 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -66,7 +66,8 @@ Supported attributes: | `attribute` | datatype | **{dotted-circle}** No | Detailed description. | | `attribute` | datatype | **{dotted-circle}** No | Detailed description. | -Response body attributes: +If successful, returns [`<status_code>`](../../api/index.md#status-codes) and the following +response attributes: | Attribute | Type | Description | |:-------------------------|:---------|:----------------------| @@ -103,7 +104,10 @@ for the section. For example: > `widget_message` [introduced](<link-to-issue>) in GitLab 14.3. ``` -## Attribute deprecation +## Deprecations + +To document the deprecation of an API endpoint, follow the steps to +[deprecate a page or topic](versions.md#deprecate-a-page-or-topic). To deprecate an attribute: @@ -121,8 +125,8 @@ To deprecate an attribute: | `widget_name` | string | **{dotted-circle}** No | [Deprecated](<link-to-issue>) in GitLab 14.7 and is planned for removal in 15.4. Use `widget_id` instead. The name of the widget. | ``` -1. Optional. To widely announce the change, or if it's a breaking change, - [update the deprecations and removals documentation](../deprecation_guidelines/#update-the-deprecations-and-removals-documentation). +To widely announce a deprecation, or if it's a breaking change, +[update the deprecations and removals documentation](../deprecation_guidelines/#update-the-deprecations-and-removals-documentation). ## Method description @@ -151,6 +155,14 @@ For information about writing attribute descriptions, see the [GraphQL API descr ## Response body description +Start the description with the following sentence, replacing `status code` with the +relevant [HTTP status code](../../api/index.md#status-codes), for example: + +```markdown +If successful, returns [`200 OK`](../../api/index.md#status-codes) and the +following response attributes: +``` + Use the following table headers to describe the response bodies. Attributes should always be in code blocks using backticks (`` ` ``). diff --git a/doc/development/documentation/site_architecture/folder_structure.md b/doc/development/documentation/site_architecture/folder_structure.md index e960a6491c7..0e8065d794f 100644 --- a/doc/development/documentation/site_architecture/folder_structure.md +++ b/doc/development/documentation/site_architecture/folder_structure.md @@ -85,6 +85,15 @@ place for it. Do not include the same information in multiple places. [Link to a single source of truth instead.](../styleguide/index.md#link-instead-of-repeating-text) +For example, if you have code in a repository other than the [primary repositories](index.md#architecture), +and documentation in the same repository, you can keep the documentation in that repository. + +Then you can either: + +- Publish it to <https://docs.gitlab.com>. +- Link to it from <https://docs.gitlab.com> by adding an entry in the global navigation. + View [an example](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/fedb6378a3c92274ba3b6031df0d34455594e4cc/content/_data/navigation.yaml#L2944). + ## References across documents - Give each folder an `index.md` page that introduces the topic, and both introduces diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index 05015fe7c5f..af24fbe303b 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -59,6 +59,19 @@ product, and all together are pulled to generate the docs website: Learn more about [the docs folder structure](folder_structure.md). +### Documentation in other repositories + +If you have code and documentation in a repository other than the [primary repositories](#architecture), +you should keep the documentation with the code in that repository. + +Then you can either: + +- [Add the repository to the list of products](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/development.md#add-a-new-product) + published at <https://docs.gitlab.com>. +- [Add an entry in the global navigation](global_nav.md#add-a-navigation-entry) for + <https://docs.gitlab.com> that links to the documentation in that repository. + View [an example](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/fedb6378a3c92274ba3b6031df0d34455594e4cc/content/_data/navigation.yaml#L2944-L2946). + ## Assets To provide an optimized site structure, design, and a search-engine friendly diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 700d64c30d1..1af0cb72055 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -49,7 +49,7 @@ GitLab adds all troubleshooting information to the documentation, no matter how unlikely a user is to encounter a situation. GitLab Support maintains their own -[troubleshooting content](../../../administration/index.md#support-team-docs) +[troubleshooting content](../../../administration/index.md#support-team-documentation) in the GitLab documentation. ### The documentation includes all media types @@ -1096,14 +1096,15 @@ copy of `https://gitlab.com/gitlab-org/gitlab`, run in a terminal: ### Animated images -Sometimes an image with animation (such as an animated GIF) -can help the reader understand a complicated interaction with the user interface. +Avoid using animated images (such as animated GIFs). They can be distracting +and annoying for users. -However, you should use them sparingly and avoid them when you can. -Do not use them to replace written descriptions of processes or the product. +If you're describing a complicated interaction in the user interface and want to +include a visual representation to help readers understand it, you can: -If you include an animated image, follow the same size and naming conventions we use for images. If the animated image loops, add at least a three -second pause to the end of the loop. +- Use a static image (screenshot) and if necessary, add callouts to emphasize an + an area of the screen. +- Create a short video of the interaction and link to it. ## Videos diff --git a/doc/development/documentation/testing.md b/doc/development/documentation/testing.md index feb10845aea..d55cbe28d9b 100644 --- a/doc/development/documentation/testing.md +++ b/doc/development/documentation/testing.md @@ -361,6 +361,7 @@ To configure Vale in your editor, install one of the following as appropriate: - Sublime Text [`SublimeLinter-contrib-vale` package](https://packagecontrol.io/packages/SublimeLinter-contrib-vale). - Visual Studio Code [`errata-ai.vale-server` extension](https://marketplace.visualstudio.com/items?itemName=errata-ai.vale-server). You can configure the plugin to [display only a subset of alerts](#show-subset-of-vale-alerts). +- Atom [`atomic-vale` package](https://atom.io/packages/atomic-vale). - Vim [ALE plugin](https://github.com/dense-analysis/ale). - JetBrains IDEs - No plugin exists, but [this issue comment](https://github.com/errata-ai/vale-server/issues/39#issuecomment-751714451) diff --git a/doc/development/event_store.md b/doc/development/event_store.md index fa7208ead04..ffde51216cf 100644 --- a/doc/development/event_store.md +++ b/doc/development/event_store.md @@ -293,6 +293,8 @@ in the `handle_event` method of the subscriber worker. ## Testing +### Testing the publisher + The publisher's responsibility is to ensure that the event is published correctly. To test that an event has been published correctly, we can use the RSpec matcher `:publish_event`: @@ -308,6 +310,25 @@ it 'publishes a ProjectDeleted event with project id and namespace id' do end ``` +It is also possible to compose matchers inside the `:publish_event` matcher. +This could be useful when we want to assert that an event is created with a certain kind of value, +but we do not know the value in advance. An example of this is when publishing an event +after creating a new record. + +```ruby +it 'publishes a ProjectCreatedEvent with project id and namespace id' do + # The project ID will only be generated when the `create_project` + # is called in the expect block. + expected_data = { project_id: kind_of(Numeric), namespace_id: group_id } + + expect { create_project(user, name: 'Project', path: 'project', namespace_id: group_id) } + .to publish_event(Projects::ProjectCreatedEvent) + .with(expected_data) +end +``` + +### Testing the subscriber + The subscriber must ensure that a published event can be consumed correctly. For this purpose we have added helpers and shared examples to standardize the way we test subscribers: diff --git a/doc/development/experiment_guide/experiment_code_reviews.md b/doc/development/experiment_guide/experiment_code_reviews.md index eda316db9d4..07bc0f59463 100644 --- a/doc/development/experiment_guide/experiment_code_reviews.md +++ b/doc/development/experiment_guide/experiment_code_reviews.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Adoption +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/experiment_guide/experiment_rollout.md b/doc/development/experiment_guide/experiment_rollout.md index ff0844f9d3c..e68520f7812 100644 --- a/doc/development/experiment_guide/experiment_rollout.md +++ b/doc/development/experiment_guide/experiment_rollout.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Adoption +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/experiment_guide/implementing_experiments.md b/doc/development/experiment_guide/implementing_experiments.md index c9e277873dc..19200d48637 100644 --- a/doc/development/experiment_guide/implementing_experiments.md +++ b/doc/development/experiment_guide/implementing_experiments.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Adoption +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/experiment_guide/index.md b/doc/development/experiment_guide/index.md index 163cd009c51..e11e516485a 100644 --- a/doc/development/experiment_guide/index.md +++ b/doc/development/experiment_guide/index.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Activation +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/experiment_guide/testing_experiments.md b/doc/development/experiment_guide/testing_experiments.md index a73896c8436..78a5d606490 100644 --- a/doc/development/experiment_guide/testing_experiments.md +++ b/doc/development/experiment_guide/testing_experiments.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Activation +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 67b53fa0299..10db332d64c 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -597,7 +597,7 @@ export default { Note that, even if the directive evaluates to `false`, the guarded entity is sent to the backend and matched against the GraphQL schema. So this approach requires that the feature-flagged entity exists in the schema, even if the feature flag is disabled. When the feature flag is turned off, it -is recommended that the resolver returns `null` at the very least. +is recommended that the resolver returns `null` at the very least using the same feature flag as the frontend. See the [API GraphQL guide](../api_graphql_styleguide.md#frontend-and-backend-feature-flag-strategies). ##### Different versions of a query @@ -617,8 +617,10 @@ export default { }; ``` -This approach is not recommended as it results in bigger merge requests and requires maintaining -two similar queries for as long as the feature flag exists. This can be used in cases where the new +##### Avoiding multiple query versions + +The multiple version approach is not recommended as it results in bigger merge requests and requires maintaining +two similar queries for as long as the feature flag exists. Multiple versions can be used in cases where the new GraphQL entities are not yet part of the schema, or if they are feature-flagged at the schema level (`new_entity: :feature_flag`). diff --git a/doc/development/fe_guide/haml.md b/doc/development/fe_guide/haml.md index 00096ce7fdc..7e72570454e 100644 --- a/doc/development/fe_guide/haml.md +++ b/doc/development/fe_guide/haml.md @@ -8,15 +8,20 @@ info: To determine the technical writer assigned to the Stage/Group associated w [HAML](https://haml.info/) is the [Ruby on Rails](https://rubyonrails.org/) template language that GitLab uses. -## GitLab UI form builder +## HAML and our Pajamas Design System [GitLab UI](https://gitlab-org.gitlab.io/gitlab-ui/) is a Vue component library that conforms -to the [Pajamas design system](https://design.gitlab.com/). Most of these components +to the [Pajamas design system](https://design.gitlab.com/). Many of these components rely on JavaScript and therefore can only be used in Vue. -However, some of the simpler components (checkboxes, radio buttons, form inputs) can be -used in HAML by applying the correct CSS classes to the elements. A custom -[Ruby on Rails form builder](https://gitlab.com/gitlab-org/gitlab/-/blob/7c108df101e86d8a27d69df2b5b1ff1fc24133c5/lib/gitlab/form_builders/gitlab_ui_form_builder.rb) exists to help use GitLab UI components in HAML. +However, some of the simpler components (such as buttons, checkboxes, or form inputs) can be +used in HAML: + +- Some of the Pajamas components are available as a [ViewComponent](view_component.md#pajamas-components). Use these when possible. +- If no ViewComponent exists, why not go ahead and create it? Talk to the Foundations team if you need help. +- As a fallback, this can be done by applying the correct CSS classes to the elements. +- A custom +[Ruby on Rails form builder](https://gitlab.com/gitlab-org/gitlab/-/blob/7c108df101e86d8a27d69df2b5b1ff1fc24133c5/lib/gitlab/form_builders/gitlab_ui_form_builder.rb) exists to help use GitLab UI components in HAML forms. ### Use the GitLab UI form builder @@ -100,7 +105,7 @@ Currently only the listed components are available but more components are plann This component supports [ViewComponent slots](https://viewcomponent.org/guide/slots.html). -| Slot | Description +| Slot | Description |---|---| | `label` | Checkbox label content. This slot can be used instead of the `label` argument. | | `help_text` | Help text content displayed below the checkbox. This slot can be used instead of the `help_text` argument. | @@ -128,7 +133,7 @@ This component supports [ViewComponent slots](https://viewcomponent.org/guide/sl This component supports [ViewComponent slots](https://viewcomponent.org/guide/slots.html). -| Slot | Description +| Slot | Description |---|---| | `label` | Checkbox label content. This slot can be used instead of the `label` argument. | | `help_text` | Help text content displayed below the radio button. This slot can be used instead of the `help_text` argument. | diff --git a/doc/development/fe_guide/index.md b/doc/development/fe_guide/index.md index 9ef4375d795..544985d7edc 100644 --- a/doc/development/fe_guide/index.md +++ b/doc/development/fe_guide/index.md @@ -89,6 +89,10 @@ How to use [GraphQL](graphql.md). How to use [HAML](haml.md). +## ViewComponent + +How we use [ViewComponent](view_component.md). + ## Icons and Illustrations How we use SVG for our [Icons and Illustrations](icons.md). diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md index 5d5b250e9a9..451b0c8a4c6 100644 --- a/doc/development/fe_guide/style/scss.md +++ b/doc/development/fe_guide/style/scss.md @@ -153,7 +153,7 @@ Usage of the `extend` at-rule is prohibited due to [memory leaks](https://gitlab } // Good -@mixing gl-pt-3 { +@mixin gl-pt-3 { padding-top: 12px; } diff --git a/doc/development/fe_guide/view_component.md b/doc/development/fe_guide/view_component.md new file mode 100644 index 00000000000..f4bb7ac3a2e --- /dev/null +++ b/doc/development/fe_guide/view_component.md @@ -0,0 +1,174 @@ +--- +stage: Ecosystem +group: Foundations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# ViewComponent + +ViewComponent is a framework for creating reusable, testable & encapsulated view +components with Ruby on Rails, without the need for a JavaScript framework like Vue. +They are rendered server-side and can be seamlessly used with template languages like [Haml](haml.md). + +Refer to the official [documentation](https://viewcomponent.org/) to learn more or +watch this [introduction video](https://youtu.be/akRhUbvtnmo). + +## Pajamas components + +Some of the components of our [Pajamas](https://design.gitlab.com) design system are +available as a ViewComponent in `app/components/pajamas`. + +NOTE: +We have a small but growing number of Pajamas components. Reach out to the +[Foundations team](https://about.gitlab.com/handbook/engineering/development/dev/ecosystem/foundations/) +if the component you are looking for is not yet available. + +### Available components + +#### Alert + +The `Pajamas::AlertComponent` follows the [Pajamas Alert](https://design.gitlab.com/components/alert) specification. + +**Examples:** + +By default this creates a dismissible info alert with icon: + +```haml += render Pajamas::AlertComponent.new(title: "Almost done!") +``` + +You can set variant, hide the icons and more: + +```haml += render Pajamas::AlertComponent.new(title: "All done!", + variant: :success, + dismissible: :false, + show_icon: false) +``` + +For the full list of options, see its +[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/components/pajamas/alert_component.rb). + +#### Banner + +The `Pajamas::BannerComponent` follows the [Pajamas Banner](https://design.gitlab.com/components/banner) specification. + +**Examples:** + +In its simplest form the banner component looks like this: + +```haml += render Pajamas::BannerComponent.new(button_text: 'Learn more', button_link: example_path, + svg_path: 'illustrations/example.svg') do |c| + - c.title { 'Hello world!' } + %p Content of your banner goes here... +``` + +If you have a need for more control, you can also use the `illustration` slot +instead of `svg_path` and the `primary_action` slot instead of `button_text` and `button_link`: + +```haml += render Pajamas::BannerComponent.new do |c| + - c.illustration do + = custom_icon('my_inline_svg') + - c.title do + Hello world! + - c.primary_action do + = render 'my_button_in_a_partial' +``` + +For the full list of options, see its +[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/components/pajamas/banner_component.rb). + +#### Button + +The `Pajamas::ButtonComponent` follows the [Pajamas Button](https://design.gitlab.com/components/button) specification. + +**Examples:** + +The button component has a lot of options but all of them have good defaults, +so the simplest button looks like this: + +```haml += render Pajamas::ButtonComponent.new do |c| + = _('Button text goes here') +``` + +The following example shows most of the available options: + +```haml += render Pajamas::ButtonComponent.new(category: :secondary, + variant: :danger, + size: :small, + type: :submit, + disabled: true, + loading: false, + block: true) do |c| + Button text goes here +``` + +You can also create button-like looking `<a>` tags, like this: + +```haml += render Pajamas::ButtonComponent.new(href: root_path) do |c| + Go home +``` + +For the full list of options, see its +[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/components/pajamas/button_component.rb). + +#### Card + +The `Pajamas::CardComponent` follows the [Pajamas Card](https://design.gitlab.com/components/card) specification. + +**Examples:** + +The card has one mandatory `body` slot and optional `header` and `footer` slots: + +```haml += render Pajamas::CardComponent.new do |c| + - c.header do + I'm the header. + - c.body do + %p Multiple line + %p body content. + - c.footer do + Footer goes here. +``` + +If you want to add custom attributes to any of these or the card itself, use the following options: + +```haml += render Pajamas::CardComponent.new(card_options: {id: "my-id"}, body_options: {data: { count: 1 }}) +``` + +`header_options` and `footer_options` are available, too. + +For the full list of options, see its +[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/components/pajamas/card_component.rb). + +#### Toggle + +The `Pajamas::ToggleComponent` follows the [Pajamas Toggle](https://design.gitlab.com/components/toggle) specification. + +```haml += render Pajamas::ToggleComponent.new(classes: 'js-force-push-toggle', + label: s_("ProtectedBranch|Toggle allowed to force push"), + is_checked: protected_branch.allow_force_push, + label_position: :hidden) + Leverage this block to render a rich help text. To render a plain text help text, prefer the `help` parameter. +``` + +NOTE: +**The toggle ViewComponent is special as it depends on the Vue.js component.** +To actually initialize this component, make sure to call the `initToggle` helper from `~/toggles`. + +For the full list of options, see its +[source](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/components/pajamas/toggle_component.rb). + +### Best practices + +- If you are about to create a new view in Haml, use the available components + over creating plain Haml tags with CSS classes. +- If you are making changes to an existing Haml view and see, for example, a + button that is still implemented with plain Haml, consider migrating it to use a ViewComponent. diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index ae13e3fc8c5..7943ae119be 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -19,8 +19,8 @@ What is described in the following sections can be found in these examples: ## Vue architecture All new features built with Vue.js must follow a [Flux architecture](https://facebook.github.io/flux/). -The main goal we are trying to achieve is to have only one data flow and only one data entry. -In order to achieve this goal we use [Vuex](#vuex). +The main goal we are trying to achieve is to have only one data flow, and only one data entry. +To achieve this goal we use [Vuex](#vuex). You can also read about this architecture in Vue documentation about [state management](https://vuejs.org/v2/guide/state-management.html#Simple-State-Management-from-Scratch) @@ -48,8 +48,8 @@ Let's look into each of them: ### An `index.js` file -This is the index file of your new feature. This is where the root Vue instance -of the new feature should be. +This file is the index file of your new feature. The root Vue instance +of the new feature should be here. The Store and the Service should be imported and initialized in this file and provided as a prop to the main component. @@ -62,17 +62,16 @@ Be sure to read about [page-specific JavaScript](performance.md#page-specific-ja While mounting a Vue application, you might need to provide data from Rails to JavaScript. To do that, you can use the `data` attributes in the HTML element and query them while mounting the application. - You should only do this while initializing the application, because the mounted element is replaced with a Vue-generated DOM. The advantage of providing data from the DOM to the Vue instance through `props` or `provide` in the `render` function, instead of querying the DOM inside the main Vue -component, is that you avoid the need to create a fixture or an HTML element in the unit test. +component, is that you avoid creating a fixture or an HTML element in the unit test. -##### provide/inject +##### `provide` and `inject` -Vue supports dependency injection through [provide/inject](https://vuejs.org/v2/api/#provide-inject). +Vue supports dependency injection through [`provide` and `inject`](https://vuejs.org/v2/api/#provide-inject). In the component the `inject` configuration accesses the values `provide` passes down. This example of a Vue app initialization shows how the `provide` configuration passes a value from HAML to the component: @@ -119,13 +118,16 @@ Using dependency injection to provide values from HAML is ideal when: - The injected value doesn't need an explicit validation against its data type or contents. - The value doesn't need to be reactive. -- There are multiple components in the hierarchy that need access to this value where +- Multiple components exist in the hierarchy that need access to this value where prop-drilling becomes an inconvenience. Prop-drilling when the same prop is passed through all components in the hierarchy until the component that is genuinely using it. -Dependency injection can potentially break a child component (either an immediate child or multiple levels deep) if the value declared in the `inject` configuration doesn't have defaults defined and the parent component has not provided the value using the `provide` configuration. +Dependency injection can potentially break a child component (either an immediate child or multiple levels deep) if both conditions are true: + +- The value declared in the `inject` configuration doesn't have defaults defined. +- The parent component has not provided the value using the `provide` configuration. -- A [default value](https://vuejs.org/guide/components/provide-inject.html#injection-default-values) might be useful in contexts where it makes sense. +A [default value](https://vuejs.org/guide/components/provide-inject.html#injection-default-values) might be useful in contexts where it makes sense. ##### props @@ -155,7 +157,8 @@ return new Vue({ }); ``` -> When adding an `id` attribute to mount a Vue application, please make sure this `id` is unique +NOTE: +When adding an `id` attribute to mount a Vue application, make sure this `id` is unique across the codebase. For more information on why we explicitly declare the data being passed into the Vue app, @@ -165,9 +168,9 @@ refer to our [Vue style guide](style/vue.md#basic-rules). When composing a form with Rails, the `name`, `id`, and `value` attributes of form inputs are generated to match the backend. It can be helpful to have access to these generated attributes when converting -a Rails form to Vue, or when [integrating components (datepicker, project selector, etc)](https://gitlab.com/gitlab-org/gitlab/-/blob/8956ad767d522f37a96e03840595c767de030968/app/assets/javascripts/access_tokens/index.js#L15) into it. +a Rails form to Vue, or when [integrating components](https://gitlab.com/gitlab-org/gitlab/-/blob/8956ad767d522f37a96e03840595c767de030968/app/assets/javascripts/access_tokens/index.js#L15) (such as a date picker or project selector) into it. The [`parseRailsFormFields`](https://gitlab.com/gitlab-org/gitlab/-/blob/fe88797f682c7ff0b13f2c2223a3ff45ada751c1/app/assets/javascripts/lib/utils/forms.js#L107) utility can be used to parse the generated form input attributes so they can be passed to the Vue application. -This allows us to easily integrate Vue components without changing how the form submits. +This enables us to integrate Vue components without changing how the form submits. ```haml -# form.html.haml @@ -245,7 +248,7 @@ export default { We query the `gl` object for data that doesn't change during the application's life cycle in the same place we query the DOM. By following this practice, we can -avoid the need to mock the `gl` object, which makes tests easier. It should be done while +avoid mocking the `gl` object, which makes tests easier. It should be done while initializing our Vue instance, and the data should be provided as `props` to the main component: ```javascript @@ -263,8 +266,8 @@ return new Vue({ #### Accessing feature flags -Use Vue's [provide/inject](https://vuejs.org/v2/api/#provide-inject) mechanism -to make feature flags available to any descendant components in a Vue +Use the [`provide` and `inject`](https://vuejs.org/v2/api/#provide-inject) mechanisms +in Vue to make feature flags available to any descendant components in a Vue application. The `glFeatures` object is already provided in `commons/vue.js`, so only the mixin is required to use the flags: @@ -303,14 +306,14 @@ This approach has a few benefits: }); ``` -- No need to access a global variable, except in the application's +- Accessing a global variable is not required, except in the application's [entry point](#accessing-the-gl-object). ### A folder for Components This folder holds all components that are specific to this new feature. -If you need to use or create a component that is likely to be used somewhere -else, please refer to `vue_shared/components`. +To use or create a component that is likely to be used somewhere +else, refer to `vue_shared/components`. A good guideline to know when you should create a component is to think if it could be reusable elsewhere. @@ -330,7 +333,7 @@ Check this [page](vuex.md) for more details. ### Mixing Vue and jQuery - Mixing Vue and jQuery is not recommended. -- If you need to use a specific jQuery plugin in Vue, [create a wrapper around it](https://vuejs.org/v2/examples/select2.html). +- To use a specific jQuery plugin in Vue, [create a wrapper around it](https://vuejs.org/v2/examples/select2.html). - It is acceptable for Vue to listen to existing jQuery events using jQuery event listeners. - It is not recommended to add new jQuery events for Vue to interact with jQuery. @@ -356,22 +359,171 @@ cannot use primitives or objects. #### Why -There are additional reasons why having a JavaScript class presents maintainability issues on a huge codebase: +Additional reasons why having a JavaScript class presents maintainability issues on a huge codebase: - After a class is created, it can be extended in a way that can infringe Vue reactivity and best practices. - A class adds a layer of abstraction, which makes the component API and its inner workings less clear. - It makes it harder to test. Because the class is instantiated by the component data function, it is harder to 'manage' component and class separately. -- Adding Object Oriented Principles (OOP) to a functional codebase adds yet another way of writing code, reducing consistency and clarity. +- Adding Object Oriented Principles (OOP) to a functional codebase adds another way of writing code, reducing consistency and clarity. ## Style guide -Please refer to the Vue section of our [style guide](style/vue.md) +Refer to the Vue section of our [style guide](style/vue.md) for best practices while writing and testing your Vue components and templates. +## Composition API + +With Vue 2.7 it is possible to use [Composition API](https://vuejs.org/guide/introduction.html#api-styles) in Vue components and as standalone composables. + +### Prefer `<script>` over `<script setup>` + +Composition API allows you to place the logic in the `<script>` section of the component or to have a dedicated `<script setup>` section. We should use `<script>` and add Composition API to components using `setup()` property: + +```html +<script> + import { computed } from 'vue'; + + export default { + name: 'MyComponent', + setup(props) { + const doubleCount = computed(() => props.count*2) + } + } +</script> +``` + +### Aim to have one API style per component + +When adding `setup()` property to Vue component, consider refactoring it to Composition API entirely. It's not always feasible, especially for large components, but we should aim to have one API style per component for readability and maintainability. + +### Composables + +With Composition API, we have a new way of abstracting logic including reactive state to _composables_. Composable is the function that can accept parameters and return reactive properties and methods to be used in Vue component. + +```javascript +// useCount.js +import { ref } from 'vue'; + +export function useCount(initialValue) { + const count = ref(initialValue) + + function incrementCount() { + ref.value += 1 + } + + function decrementCount() { + ref.value -= 1 + } + + return { count, incrementCount, decrementCount } +} +``` + +```javascript +// MyComponent.vue +import { useCount } from 'useCount' + +export default { + name: 'MyComponent', + setup() { + const { count, incrementCount, decrementCount } = useCount(5) + + return { count, incrementCount, decrementCount } + } +} +``` + +#### Prefix function and file names with `use` + +Common naming convention in Vue for composables is to prefix them with `use` and then refer to composable functionality briefly (`useBreakpoints`, `useGeolocation` etc). The same rule applies to the `.js` files containing composables - they should start with `use_` even if the file contains more than one composable. + +#### Avoid lifecycle pitfalls + +When building a composable, we should aim to keep it as simple as possible. Lifecycle hooks add complexity to composables and might lead to unexpected side effects. To avoid that we should follow these principles: + +- Minimize lifecycle hooks usage whenever possible, prefer accepting/returning callbacks instead. +- If your composable needs lifecycle hooks, make sure it also performs a cleanup. If we add a listener on `onMounted`, we should remove it on `onUnmounted` within the same composable. +- Always set up lifecycle hooks immediately: + +```javascript +// bad +const useAsyncLogic = () => { + const action = async () => { + await doSomething(); + onMounted(doSomethingElse); + }; + return { action }; +}; + +// OK +const useAsyncLogic = () => { + const done = ref(false); + onMounted(() => { + watch( + done, + () => done.value && doSomethingElse(), + { immediate: true }, + ); + }); + const action = async () => { + await doSomething(); + done.value = true; + }; + return { action }; +}; +``` + +#### Avoid escape hatches + +It might be tempting to write a composable that does everything as a black box, using some of the escape hatches that Vue provides. But for most of the cases this makes them too complex and hard to maintain. One escape hatch is the `getCurrentInstance` method. This method returns an instance of a current rendering component. Instead of using that method, you should prefer passing down the data or methods to a composable via arguments. + +```javascript +const useSomeLogic = () => { + doSomeLogic(); + getCurrentInstance().emit('done'); // bad +}; +``` + +```javascript +const done = () => emit('done'); + +const useSomeLogic = (done) => { + doSomeLogic(); + done(); // good, composable doesn't try to be too smart +} +``` + +#### Composables and Vuex + +We should always prefer to avoid using Vuex state in composables. In case it's not possible, we should use props to receive that state, and emit events from the `setup` to update the Vuex state. A parent component should be responsible to get that state from Vuex, and mutate it on events emitted from a child. You should **never mutate a state that's coming down from a prop**. If a composable must mutate a Vuex state, it should use a callback to emit an event. + +```javascript +const useAsyncComposable = ({ state, update }) => { + const start = async () => { + const newState = await doSomething(state); + update(newState); + }; + return { start }; +}; + +const ComponentWithComposable = { + setup(props, { emit }) { + const update = (data) => emit('update', data); + const state = computed(() => props.state); // state from Vuex + const { start } = useAsyncComposable({ state, update }); + start(); + }, +}; +``` + +#### Testing composables + +<!-- TBD --> + ## Testing Vue Components -Please refer to the [Vue testing style guide](style/vue.md#vue-testing) +Refer to the [Vue testing style guide](style/vue.md#vue-testing) for guidelines and best practices for testing your Vue components. Each Vue component has a unique output. This output is always present in the render function. @@ -500,8 +652,8 @@ component under test, with the `computed` property, for example). Remember to us ### Events -We should test for events emitted in response to an action in our component. This is used to -verify the correct events are being fired with the correct arguments. +We should test for events emitted in response to an action in our component. This testing +verifies the correct events are being fired with the correct arguments. For any DOM events we should use [`trigger`](https://v1.test-utils.vuejs.org/api/wrapper/#trigger) to fire out event. @@ -519,8 +671,7 @@ it('should fire the click event', () => { }) ``` -When we need to fire a Vue event, we should use [`emit`](https://vuejs.org/v2/guide/components-custom-events.html) -to fire our event. +When firing a Vue event, use [`emit`](https://vuejs.org/v2/guide/components-custom-events.html). ```javascript wrapper = shallowMount(DropdownItem); diff --git a/doc/development/feature_development.md b/doc/development/feature_development.md new file mode 100644 index 00000000000..a5d74a0bfd9 --- /dev/null +++ b/doc/development/feature_development.md @@ -0,0 +1,197 @@ +--- +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +--- + +# Feature development + +Consult these topics for information on contributing to specific GitLab features. + +## UX and Frontend guides + +- [GitLab Design System](https://design.gitlab.com/), for building GitLab with + existing CSS styles and elements +- [Frontend guidelines](fe_guide/index.md) +- [Emoji guide](fe_guide/emojis.md) + +## Backend guides + +### General + +- [Directory structure](directory_structure.md) +- [GitLab EventStore](event_store.md) to publish/subscribe to domain events +- [GitLab utilities](utilities.md) +- [Newlines style guide](newlines_styleguide.md) +- [Logging](logging.md) +- [Dealing with email/mailers](emails.md) +- [Kubernetes integration guidelines](kubernetes.md) +- [Permissions](permissions.md) +- [Code comments](code_comments.md) +- [Windows Development on GCP](windows.md) +- [FIPS compliance](fips_compliance.md) +- [`Gemfile` guidelines](gemfile.md) +- [Ruby upgrade guidelines](ruby_upgrade.md) + +### Things to be aware of + +- [Gotchas](gotchas.md) to avoid +- [Avoid modules with instance variables](module_with_instance_variables.md), if + possible +- [Guidelines for reusing abstractions](reusing_abstractions.md) +- [Ruby 3 gotchas](ruby3_gotchas.md) + +### Rails Framework related + +- [Routing](routing.md) +- [Rails initializers](rails_initializers.md) +- [Mass Inserting Models](mass_insert.md) +- [Issuable-like Rails models](issuable-like-models.md) +- [Issue types vs first-class types](issue_types.md) +- [DeclarativePolicy framework](policies.md) +- [Rails update guidelines](rails_update.md) + +### Debugging + +- [Pry debugging](pry_debugging.md) +- [Sidekiq debugging](../administration/troubleshooting/sidekiq.md) + +### Git specifics + +- [How Git object deduplication works in GitLab](git_object_deduplication.md) +- [Git LFS](lfs.md) + +### API + +- [API style guide](api_styleguide.md) for contributing to the API +- [GraphQL API style guide](api_graphql_styleguide.md) for contributing to the + [GraphQL API](../api/graphql/index.md) + +### GitLab components and features + +- [Developing against interacting components or features](interacting_components.md) +- [Manage feature flags](feature_flags/index.md) +- [Licensed feature availability](licensed_feature_availability.md) +- [Accessing session data](session.md) +- [How to dump production data to staging](db_dump.md) +- [Geo development](geo.md) +- [Redis guidelines](redis.md) + - [Adding a new Redis instance](redis/new_redis_instance.md) +- [Sidekiq guidelines](sidekiq/index.md) for working with Sidekiq workers +- [Working with Gitaly](gitaly.md) +- [Elasticsearch integration docs](elasticsearch.md) +- [Working with merge request diffs](diffs.md) +- [Approval Rules](approval_rules.md) +- [Repository mirroring](repository_mirroring.md) +- [Uploads development guide](uploads/index.md) +- [Auto DevOps development guide](auto_devops.md) +- [Renaming features](renaming_features.md) +- [Code Intelligence](code_intelligence/index.md) +- [Feature categorization](feature_categorization/index.md) +- [Wikis development guide](wikis.md) +- [Image scaling guide](image_scaling.md) +- [Cascading Settings](cascading_settings.md) +- [Shell commands](shell_commands.md) in the GitLab codebase +- [Value Stream Analytics development guide](value_stream_analytics.md) +- [Application limits](application_limits.md) + +### Import and Export + +- [Working with the GitHub importer](github_importer.md) +- [Import/Export development documentation](import_export.md) +- [Test Import Project](import_project.md) +- [Group migration](bulk_import.md) +- [Export to CSV](export_csv.md) + +## Performance guides + +- [Performance guidelines](performance.md) for writing code, benchmarks, and + certain patterns to avoid. +- [Caching guidelines](caching.md) for using caching in Rails under a GitLab environment. +- [Merge request performance guidelines](merge_request_performance_guidelines.md) + for ensuring merge requests do not negatively impact GitLab performance +- [Profiling](profiling.md) a URL or tracking down N+1 queries using Bullet. +- [Cached queries guidelines](cached_queries.md), for tracking down N+1 queries + masked by query caching, memory profiling and why should we avoid cached + queries. + +## Database guides + +See [database guidelines](database/index.md). + +## Integration guides + +- [Integrations development guide](integrations/index.md) +- [Jira Connect app](integrations/jira_connect.md) +- [Security Scanners](integrations/secure.md) +- [Secure Partner Integration](integrations/secure_partner_integration.md) +- [How to run Jenkins in development environment](integrations/jenkins.md) +- [How to run local `Codesandbox` integration for Web IDE Live Preview](integrations/codesandbox.md) + +## Testing guides + +- [Testing standards and style guidelines](testing_guide/index.md) +- [Frontend testing standards and style guidelines](testing_guide/frontend_testing.md) + +## Refactoring guides + +- [Refactoring guidelines](refactoring_guide/index.md) + +## Deprecation guides + +- [Deprecation guidelines](deprecation_guidelines/index.md) + +## Documentation guides + +- [Writing documentation](documentation/index.md) +- [Documentation style guide](documentation/styleguide/index.md) +- [Markdown](../user/markdown.md) + +## Internationalization (i18n) guides + +- [Introduction](i18n/index.md) +- [Externalization](i18n/externalization.md) +- [Translation](i18n/translation.md) + +## Product Intelligence guides + +- [Product Intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) +- [Service Ping guide](service_ping/index.md) +- [Snowplow guide](snowplow/index.md) + +## Experiment guide + +- [Introduction](experiment_guide/index.md) + +## Build guides + +- [Building a package for testing purposes](build_test_package.md) + +## Compliance + +- [Licensing](licensing.md) for ensuring license compliance + +## Domain-specific guides + +- [CI/CD development documentation](cicd/index.md) + +## Technical Reference by Group + +- [Create: Source Code BE](backend/create_source_code_be/index.md) + +## Other development guides + +- [Defining relations between files using projections](projections.md) +- [Reference processing](reference_processing.md) +- [Compatibility with multiple versions of the application running at the same time](multi_version_compatibility.md) +- [Features inside `.gitlab/`](features_inside_dot_gitlab.md) +- [Dashboards for stage groups](stage_group_observability/index.md) +- [Preventing transient bugs](transient/prevention-patterns.md) +- [GitLab Application SLIs](application_slis/index.md) +- [Spam protection and CAPTCHA development guide](spam_protection_and_captcha/index.md) + +## Other GitLab Development Kit (GDK) guides + +- [Run full Auto DevOps cycle in a GDK instance](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/auto_devops.md) +- [Using GitLab Runner with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/runner.md) +- [Using the Web IDE terminal with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/web_ide_terminal_gdk_setup.md) diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index d21a46142a2..140d5f826cf 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -236,6 +236,11 @@ command. For example: /chatops run feature list --staging ``` +## Toggle a feature flag + +See [rolling out changes](controls.md#rolling-out-changes) for more information about toggling +feature flags. + ## Delete a feature flag See [cleaning up feature flags](controls.md#cleaning-up) for more information about @@ -520,6 +525,8 @@ Feature.remove(:feature_flag_name) ``` - Any change behind a feature flag that is **enabled** by default **should** have a changelog entry. +- The changelog for a feature flag should describe the feature and not the + flag, unless a default on feature flag is removed keeping the new code (`other` in the flowchart above). ## Feature flags in tests diff --git a/doc/development/fips_compliance.md b/doc/development/fips_compliance.md index 5b6f6ba0d98..6261b2fda6f 100644 --- a/doc/development/fips_compliance.md +++ b/doc/development/fips_compliance.md @@ -25,26 +25,37 @@ mean FIPS 140-2. ## Current status -GitLab Inc has not committed to making GitLab FIPS-compliant at this time. We are -performing initial investigations to see how much work such an effort would be. - Read [Epic &5104](https://gitlab.com/groups/gitlab-org/-/epics/5104) for more information on the status of the investigation. -## FIPS compliance at GitLab - -In a FIPS context, compliance is a form of self-certification - if we say we are -"FIPS compliant", we mean that we *believe* we are. There are no external -certifications to acquire, but if we are aware of non-compliant areas -in GitLab, we cannot self-certify in good faith. +GitLab is actively working towards FIPS compliance. -The known areas of non-compliance are tracked in [Epic &5104](https://gitlab.com/groups/gitlab-org/-/epics/5104). +## FIPS compliance at GitLab To be compliant, all components (GitLab itself, Gitaly, etc) must be compliant, along with the communication between those components, and any storage used by them. Where functionality cannot be brought into compliance, it must be disabled when FIPS mode is enabled. +### Leveraged Cryptographic modules + +| Cryptographic module name | CMVP number | Instance type | Software component used | +|----------------------------------------------------------|-------------------------------------------------------------------------------------------------|---------------|-------------------------| +| Ubuntu 20.04 AWS Kernel Crypto API Cryptographic Module | [4132](https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/4132) | EC2 | Linux kernel | +| Ubuntu 20.04 OpenSSL Cryptographic Module | [3966](https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3966) | EC2 | Gitaly, Rails (Puma/Sidekiq) | +| Ubuntu 20.04 Libgcrypt Cryptographic Module | [3902](https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3902) | EC2 instances | `gpg`, `sshd` | +| Amazon Linux 2 Kernel Crypto API Cryptographic Module | [3709](https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3709) | EKS nodes | Linux kernel | +| Amazon Linux 2 OpenSSL Cryptographic Module | [3553](https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3553) | EKS nodes | NGINX | +| RedHat Enterprise Linux 8 OpenSSL Cryptographic Module | [3852](https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3852) | EKS nodes | UBI containers: Workhorse, Pages, Container Registry, Rails (Puma/Sidekiq), Security Analyzers | +| RedHat Enterprise Linux 8 Libgcrypt Cryptographic Module | [3784](https://csrc.nist.gov/projects/cryptographic-module-validation-program/certificate/3784) | EKS nodes | UBI containers: GitLab Shell, `gpg` | + +### Supported Operating Systems + +The supported hybrid environments are: + +- Omnibus: Ubuntu 20.04 FIPS +- EKS: Amazon Linux 2 + ## FIPS validation at GitLab Unlike FIPS compliance, FIPS validation is a formal declaration of compliance by @@ -55,81 +66,24 @@ A list of FIPS-validated modules can be found at the NIST (National Institute of Standards and Technology) [cryptographic module validation program](https://csrc.nist.gov/projects/cryptographic-module-validation-program/validated-modules). -## Setting up a FIPS-enabled development environment - -The simplest approach is to set up a virtual machine running -[Red Hat Enterprise Linux 8](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening#switching-the-system-to-fips-mode_using-the-system-wide-cryptographic-policies). - -Red Hat provide free licenses to developers, and permit the CD image to be -downloaded from the [Red Hat developer's portal](https://developers.redhat.com). -Registration is required. - -After the virtual machine is set up, you can follow the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) -installation instructions, including the [advanced instructions for RHEL](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/advanced.md#red-hat-enterprise-linux). -Note that `asdf` is not used for dependency management because it's essential to -use the RedHat-provided Go compiler and other system dependencies. - -### Enable FIPS mode - -After GDK and its dependencies are installed, run this command (as -root) and restart the virtual machine: - -```shell -fips-mode-setup --enable -``` - -You can check whether it's taken effect by running: - -```shell -fips-mode-setup --check -``` - -In this environment, OpenSSL refuses to perform cryptographic operations -forbidden by the FIPS standards. This enables you to reproduce FIPS-related bugs, -and validate fixes. - -You should be able to open a web browser inside the virtual machine and log in -to the GitLab instance. - -You can disable FIPS mode again by running this command, then restarting the -virtual machine: - -```shell -fips-mode-setup --disable -``` - -#### Detect FIPS enablement in code - -You can query `Gitlab::FIPS` in Ruby code to determine if the instance is FIPS-enabled: - -```ruby -def default_min_key_size(name) - if Gitlab::FIPS.enabled? - Gitlab::SSHPublicKey.supported_sizes(name).select(&:positive?).min || -1 - else - 0 - end -end -``` +## Install GitLab with FIPS compliance -## Nightly Omnibus FIPS builds +This guide is specifically for public users or GitLab team members with a requirement +to run a production instance of GitLab that is FIPS compliant. This guide outlines +a hybrid deployment using elements from both Omnibus and our Cloud Native GitLab installations. -The Distribution team has created [nightly FIPS Omnibus builds](https://packages.gitlab.com/gitlab/nightly-fips-builds). These -GitLab builds are compiled to use the system OpenSSL instead of the Omnibus-embedded version of OpenSSL. +### Prerequisites -See [the section on how FIPS builds are created](#how-fips-builds-are-created). +- Amazon Web Services account. Our first target environment is running on AWS, and uses other FIPS Compliant AWS resources. +- Ability to run Ubuntu 20.04 machines for GitLab. Our first target environment uses the hybrid architecture. -## Runner - -See the [documentation on installing a FIPS-compliant GitLab Runner](https://docs.gitlab.com/runner/install/#fips-compliant-gitlab-runner). - -## Set up a FIPS-enabled cluster +### Set up a FIPS-enabled cluster You can use the [GitLab Environment Toolkit](https://gitlab.com/gitlab-org/gitlab-environment-toolkit) to spin -up a FIPS-enabled cluster for development and testing. These instructions use Amazon Web Services (AWS) -because that is the first target environment, but you can adapt them for other providers. +up a FIPS-enabled cluster for development and testing. As mentioned in the prerequisites, these instructions use Amazon Web Services (AWS) +because that is the first target environment. -### Set up your environment +#### Set up your environment To get started, your AWS account must subscribe to a FIPS-enabled Amazon Machine Image (AMI) in the [AWS Marketplace console](https://aws.amazon.com/premiumsupport/knowledge-center/launch-ec2-marketplace-subscription/). @@ -138,13 +92,13 @@ This example assumes that the `Ubuntu Pro 20.04 FIPS LTS` AMI by `Canonical Group Limited` has been added your account. This operating system is used for virtual machines running in Amazon EC2. -### Omnibus +#### Omnibus The simplest way to get a FIPS-enabled GitLab cluster is to use an Omnibus reference architecture. See the [GET Quick Start Guide](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/docs/environment_quick_start_guide.md) for more details. The following instructions build on the Quick Start and are also necessary for [Cloud Native Hybrid](#cloud-native-hybrid) installations. -#### Terraform: Use a FIPS AMI +##### Terraform: Use a FIPS AMI 1. Follow the guide to set up Terraform and Ansible. 1. After [step 2b](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/-/blob/main/docs/environment_quick_start_guide.md#2b-setup-config), @@ -189,7 +143,7 @@ an instance, this would result in data loss: not only would disks be destroyed, but also GitLab secrets would be lost. There is a [Terraform lifecycle rule](https://gitlab.com/gitlab-org/gitlab-environment-toolkit/blob/2aaeaff8ac8067f23cd7b6bb5bf131061649089d/terraform/modules/gitlab_aws_instance/main.tf#L40) to ignore AMI changes. -#### Ansible: Specify the FIPS Omnibus builds +##### Ansible: Specify the FIPS Omnibus builds The standard Omnibus GitLab releases build their own OpenSSL library, which is not FIPS-validated. However, we have nightly builds that create Omnibus packages @@ -203,11 +157,11 @@ in this way: all: vars: ... - gitlab_repo_script_url: "https://packages.gitlab.com/install/repositories/gitlab/nightly-fips-builds/script.deb.sh" + gitlab_repo_script_url: "https://packages.gitlab.com/install/repositories/gitlab/gitlab-fips/script.deb.sh" gitlab_edition: "gitlab-fips" ``` -### Cloud Native Hybrid +#### Cloud Native Hybrid A Cloud Native Hybrid install uses both Omnibus and Cloud Native GitLab (CNG) images. The previous instructions cover the Omnibus part, but two @@ -216,7 +170,7 @@ additional steps are needed to enable FIPS in CNG: 1. Use a custom Amazon Elastic Kubernetes Service (EKS) AMI. 1. Use GitLab containers built with RedHat's Universal Base Image (UBI). -#### Build a custom EKS AMI +##### Build a custom EKS AMI Because Amazon does not yet publish a FIPS-enabled AMI, you have to build one yourself with Packer. @@ -259,7 +213,7 @@ be different. Building a RHEL-based system with FIPS enabled should be possible, but there is [an outstanding issue preventing the Packer build from completing](https://github.com/aws-samples/amazon-eks-custom-amis/issues/51). -#### Terraform: Use a custom EKS AMI +##### Terraform: Use a custom EKS AMI Now you can set the custom EKS AMI. @@ -286,7 +240,7 @@ Now you can set the custom EKS AMI. } ``` -#### Ansible: Use UBI images +##### Ansible: Use UBI images CNG uses a Helm Chart to manage which container images to deploy. By default, GET deploys the latest released versions that use Debian-based containers. @@ -396,6 +350,107 @@ gitlab: tag: v15.1.0-fips ``` +## FIPS Performance Benchmarking + +The Quality Engineering Enablement team assists these efforts by checking if FIPS-enabled environments perform well compared to non-FIPS environments. + +Testing shows an impact in some places, such as Gitaly SSL, but it's not large enough to impact customers. + +You can find more information on FIPS performance benchmarking in the following issue: + +- [Benchmark performance of FIPS reference architecture](https://gitlab.com/gitlab-org/gitlab/-/issues/364051#note_1010450415) + +## Setting up a FIPS-enabled development environment + +The simplest approach is to set up a virtual machine running +[Red Hat Enterprise Linux 8](https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/security_hardening/using-the-system-wide-cryptographic-policies_security-hardening#switching-the-system-to-fips-mode_using-the-system-wide-cryptographic-policies). + +Red Hat provide free licenses to developers, and permit the CD image to be +downloaded from the [Red Hat developer's portal](https://developers.redhat.com). +Registration is required. + +After the virtual machine is set up, you can follow the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit) +installation instructions, including the [advanced instructions for RHEL](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/advanced.md#red-hat-enterprise-linux). +Note that `asdf` is not used for dependency management because it's essential to +use the RedHat-provided Go compiler and other system dependencies. + +### Enable FIPS mode + +After GDK and its dependencies are installed, run this command (as +root) and restart the virtual machine: + +```shell +fips-mode-setup --enable +``` + +You can check whether it's taken effect by running: + +```shell +fips-mode-setup --check +``` + +In this environment, OpenSSL refuses to perform cryptographic operations +forbidden by the FIPS standards. This enables you to reproduce FIPS-related bugs, +and validate fixes. + +You should be able to open a web browser inside the virtual machine and log in +to the GitLab instance. + +You can disable FIPS mode again by running this command, then restarting the +virtual machine: + +```shell +fips-mode-setup --disable +``` + +#### Detect FIPS enablement in code + +You can query `Gitlab::FIPS` in Ruby code to determine if the instance is FIPS-enabled: + +```ruby +def default_min_key_size(name) + if Gitlab::FIPS.enabled? + Gitlab::SSHPublicKey.supported_sizes(name).select(&:positive?).min || -1 + else + 0 + end +end +``` + +#### Unsupported features in FIPS mode + +Some GitLab features may not work when FIPS mode is enabled. The following features +are known to not work in FIPS mode. However, there may be additional features not +listed here that also do not work properly in FIPS mode: + +- [Container Scanning](../user/application_security/container_scanning/index.md) support for scanning images in repositories that require authentication. +- [Code Quality](../ci/testing/code_quality.md) does not support operating in FIPS-compliant mode. +- [Dependency scanning](../user/application_security/dependency_scanning/index.md) support for Gradle. +- [Dynamic Application Security Testing (DAST)](../user/application_security/dast/index.md) + does not support operating in FIPS-compliant mode. +- [License compliance](../user/compliance/license_compliance/index.md). +- [Solutions for vulnerabilities](../user/application_security/vulnerabilities/index.md#resolve-a-vulnerability) + for yarn projects. +- [Static Application Security Testing (SAST)](../user/application_security/sast/index.md) + supports a reduced set of [analyzers](../user/application_security/sast/#fips-enabled-images) + when operating in FIPS-compliant mode. + +Additionally, these package repositories are disabled in FIPS mode: + +- [Conan package repository](../user/packages/conan_repository/index.md). +- [Debian package repository](../user/packages/debian_repository/index.md). + +## Nightly Omnibus FIPS builds + +The Distribution team has created [nightly FIPS Omnibus builds](https://packages.gitlab.com/gitlab/nightly-fips-builds). These +GitLab builds are compiled to use the system OpenSSL instead of the Omnibus-embedded version of OpenSSL. + +See [the section on how FIPS builds are created](#how-fips-builds-are-created). + +## Runner + +See the [documentation on installing a FIPS-compliant GitLab Runner](https://docs.gitlab.com/runner/install/#fips-compliant-gitlab-runner). + ## Verify FIPS The following sections describe ways you can verify if FIPS is enabled. diff --git a/doc/development/foreign_keys.md b/doc/development/foreign_keys.md index 77df6fbfb0d..e0dd0fe8e7c 100644 --- a/doc/development/foreign_keys.md +++ b/doc/development/foreign_keys.md @@ -28,9 +28,80 @@ Guide](migration_style_guide.md) for more information. Keep in mind that you can only safely add foreign keys to existing tables after you have removed any orphaned rows. The method `add_concurrent_foreign_key` -does not take care of this so you need to do so manually. See +does not take care of this so you must do so manually. See [adding foreign key constraint to an existing column](database/add_foreign_key_to_existing_column.md). +## Updating Foreign Keys In Migrations + +Sometimes a foreign key constraint must be changed, preserving the column +but updating the constraint condition. For example, moving from +`ON DELETE CASCADE` to `ON DELETE SET NULL` or vice-versa. + +PostgreSQL does not prevent you from adding overlapping foreign keys. It +honors the most recently added constraint. This allows us to replace foreign keys without +ever losing foreign key protection on a column. + +To replace a foreign key: + +1. [Add the new foreign key without validation](database/add_foreign_key_to_existing_column.md#prevent-invalid-records) + + The name of the foreign key constraint must be changed to add a new + foreign key before removing the old one. + + ```ruby + class ReplaceFkOnPackagesPackagesProjectId < Gitlab::Database::Migration[2.0] + disable_ddl_transaction! + + NEW_CONSTRAINT_NAME = 'fk_new' + + def up + add_concurrent_foreign_key(:packages_packages, :projects, column: :project_id, on_delete: :nullify, validate: false, name: NEW_CONSTRAINT_NAME) + end + + def down + with_lock_retries do + remove_foreign_key_if_exists(:packages_packages, column: :project_id, on_delete: :nullify, name: NEW_CONSTRAINT_NAME) + end + end + end + ``` + +1. [Validate the new foreign key](database/add_foreign_key_to_existing_column.md#validate-the-foreign-key) + + ```ruby + class ValidateFkNew < Gitlab::Database::Migration[2.0] + NEW_CONSTRAINT_NAME = 'fk_new' + + # foreign key added in <link to MR or path to migration adding new FK> + def up + validate_foreign_key(:packages_packages, name: NEW_CONSTRAINT_NAME) + end + + def down + # no-op + end + end + ``` + +1. Remove the old foreign key: + + ```ruby + class RemoveFkOld < Gitlab::Database::Migration[2.0] + OLD_CONSTRAINT_NAME = 'fk_old' + + # new foreign key added in <link to MR or path to migration adding new FK> + # and validated in <link to MR or path to migration validating new FK> + def up + remove_foreign_key_if_exists(:packages_packages, column: :project_id, on_delete: :cascade, name: OLD_CONSTRAINT_NAME) + end + + def down + # Validation is skipped here, so if rolled back, this will need to be revalidated in a separate migration + add_concurrent_foreign_key(:packages_packages, :projects, column: :project_id, on_delete: :cascade, validate: false, name: OLD_CONSTRAINT_NAME) + end + end + ``` + ## Cascading Deletes Every foreign key must define an `ON DELETE` clause, and in 99% of the cases diff --git a/doc/development/geo.md b/doc/development/geo.md index 18dffe42177..9e9bd85ecd8 100644 --- a/doc/development/geo.md +++ b/doc/development/geo.md @@ -19,11 +19,25 @@ Geo handles replication for different components: - [Database](#database-replication): includes the entire application, except cache and jobs. - [Git repositories](#repository-replication): includes both projects and wikis. -- [Uploaded blobs](#uploads-replication): includes anything from images attached on issues +- [Blobs](#blob-replication): includes anything from images attached on issues to raw logs and assets from CI. With the exception of the Database replication, on a *secondary* site, everything is coordinated -by the [Geo Log Cursor](#geo-log-cursor). +by the [Geo Log Cursor](#geo-log-cursor-daemon). + +### Replication states + +The following diagram illustrates how the replication works. Some allowed transitions are omitted for clarity. + +```mermaid +stateDiagram-v2 + Pending --> Started + Started --> Synced + Started --> Failed + Synced --> Pending: Mark for resync + Failed --> Pending: Mark for resync + Failed --> Started: Retry +``` ### Geo Log Cursor daemon @@ -66,7 +80,7 @@ the state of every repository in the [tracking database](#tracking-database). There are a few ways a repository gets replicated by the: - [Repository Sync worker](#repository-sync-worker). -- [Geo Log Cursor](#geo-log-cursor). +- [Geo Log Cursor](#geo-log-cursor-daemon). #### Project Registry @@ -97,26 +111,211 @@ projects that need updating. Those projects can be: timestamp that is more recent than the `last_repository_successful_sync_at` timestamp in the `Geo::ProjectRegistry` model. - Manual: The administrator can manually flag a repository to resync in the - [Geo Admin Area](../user/admin_area/geo_nodes.md). + [Geo Admin Area](../user/admin_area/geo_sites.md). When we fail to fetch a repository on the secondary `RETRIES_BEFORE_REDOWNLOAD` times, Geo does a so-called _re-download_. It will do a clean clone into the `@geo-temporary` directory in the root of the storage. When it's successful, we replace the main repository with the newly cloned one. -### Uploads replication +### Blob replication + +Blobs such as [uploads](uploads/index.md), LFS objects, and CI job artifacts, are replicated to the **secondary** site with the [Self-Service Framework](geo/framework.md). To track the state of syncing, each model has a corresponding registry table, for example `Upload` has `Geo::UploadRegistry` in the [PostgreSQL Geo Tracking Database](#tracking-database). -File uploads are also being replicated to the **secondary** site. To -track the state of syncing, the `Geo::UploadRegistry` model is used. +#### Blob replication happy path workflows between services + +Job artifacts are used in the diagrams below, as one example of a blob. + +##### Replicating a new job artifact + +Primary site: + +```mermaid +sequenceDiagram + participant R as Runner + participant P as Puma + participant DB as PostgreSQL + participant SsP as Secondary site PostgreSQL + R->>P: Upload artifact + P->>DB: Insert `ci_job_artifacts` row + P->>DB: Insert `geo_events` row + P->>DB: Insert `geo_event_log` row + DB->>SsP: Replicate rows +``` -#### Upload Registry +- A [Runner](https://docs.gitlab.com/runner/) uploads an artifact +- [Puma](architecture.md#puma) inserts `ci_job_artifacts` row +- Puma inserts `geo_events` row with data like "Job Artifact with ID 123 was updated" +- Puma inserts `geo_event_log` row pointing to the `geo_events` row (because we built SSF on top of some legacy logic) +- [PostgreSQL](architecture.md#postgresql) streaming replication inserts the rows in the read replica -Similar to the [Project Registry](#project-registry), there is a -`Geo::UploadRegistry` model that tracks the synced uploads. +Secondary site, after the PostgreSQL DB rows have been replicated: + +```mermaid +sequenceDiagram + participant DB as PostgreSQL + participant GLC as Geo Log Cursor + participant R as Redis + participant S as Sidekiq + participant TDB as PostgreSQL Tracking DB + participant PP as Primary site Puma + GLC->>DB: Query `geo_event_log` + GLC->>DB: Query `geo_events` + GLC->>R: Enqueue `Geo::EventWorker` + S->>R: Pick up `Geo::EventWorker` + S->>TDB: Insert to `job_artifact_registry`, "starting sync" + S->>PP: GET <primary site internal URL>/geo/retrieve/job_artifact/123 + S->>TDB: Update `job_artifact_registry`, "synced" +``` + +- [Geo Log Cursor](#geo-log-cursor-daemon) loop finds the new `geo_event_log` row +- Geo Log Cursor processes the `geo_events` row + - Geo Log Cursor enqueues `Geo::EventWorker` job passing through the `geo_events` row data +- [Sidekiq](architecture.md#sidekiq) picks up `Geo::EventWorker` job + - Sidekiq inserts `job_artifact_registry` row in the [PostgreSQL Geo Tracking Database](#tracking-database) because it doesn't exist, and marks it "started sync" + - Sidekiq does a GET request on an API endpoint at the primary Geo site and downloads the file + - Sidekiq marks the `job_artifact_registry` row as "synced" and "pending verification" + +##### Backfilling existing job artifacts + +- Sysadmin has an existing GitLab site without Geo +- There are existing CI jobs and job artifacts +- Sysadmin sets up a new GitLab site and configures it to be a secondary Geo site + +Secondary site: + +There are two cronjobs running every minute: `Geo::Secondary::RegistryConsistencyWorker` and `Geo::RegistrySyncWorker`. The workflow below is split into two, along those lines. + +```mermaid +sequenceDiagram + participant SC as Sidekiq-cron + participant R as Redis + participant S as Sidekiq + participant DB as PostgreSQL + participant TDB as PostgreSQL Tracking DB + SC->>R: Enqueue `Geo::Secondary::RegistryConsistencyWorker` + S->>R: Pick up `Geo::Secondary::RegistryConsistencyWorker` + S->>DB: Query `ci_job_artifacts` + S->>TDB: Query `job_artifact_registry` + S->>TDB: Insert to `job_artifact_registry` +``` -CI Job Artifacts and LFS objects are synced in a similar way as uploads, -but they are tracked by `Geo::JobArtifactRegistry`, and `Geo::LfsObjectRegistry` -models respectively. +- [Sidekiq-cron](https://github.com/ondrejbartas/sidekiq-cron) enqueues a `Geo::Secondary::RegistryConsistencyWorker` job every minute. As long as it is actively doing work (creating and deleting rows), this job immediately reenqueues itself. This job uses an exclusive lease to prevent multiple instances of itself from running simultaneously. +- [Sidekiq](architecture.md#sidekiq) picks up `Geo::Secondary::RegistryConsistencyWorker` job + - Sidekiq queries `ci_job_artifacts` table for up to 10000 rows + - Sidekiq queries `job_artifact_registry` table for up to 10000 rows + - Sidekiq inserts a `job_artifact_registry` row in the [PostgreSQL Geo Tracking Database](#tracking-database) corresponding to the existing Job Artifact + +```mermaid +sequenceDiagram + participant SC as Sidekiq-cron + participant R as Redis + participant S as Sidekiq + participant DB as PostgreSQL + participant TDB as PostgreSQL Tracking DB + participant PP as Primary site Puma + SC->>R: Enqueue `Geo::RegistrySyncWorker` + S->>R: Pick up `Geo::RegistrySyncWorker` + S->>TDB: Query `*_registry` tables + S->>R: Enqueue `Geo::EventWorker`s + S->>R: Pick up `Geo::EventWorker` + S->>TDB: Insert to `job_artifact_registry`, "starting sync" + S->>PP: GET <primary site internal URL>/geo/retrieve/job_artifact/123 + S->>TDB: Update `job_artifact_registry`, "synced" +``` + +- [Sidekiq-cron](https://github.com/ondrejbartas/sidekiq-cron) enqueues a `Geo::RegistrySyncWorker` job every minute. As long as it is actively doing work, this job loops for up to an hour scheduling sync jobs. This job uses an exclusive lease to prevent multiple instances of itself from running simultaneously. +- [Sidekiq](architecture.md#sidekiq) picks up `Geo::RegistrySyncWorker` job + - Sidekiq queries all `registry` tables in the [PostgreSQL Geo Tracking Database](#tracking-database) for "never attempted sync" rows. It interleaves rows from each table and adds them to an in-memory queue. + - If the previous step yielded less than 1000 rows, then Sidekiq queries all `registry` tables for "failed sync and ready to retry" rows and interleaves those and adds them to the in-memory queue. + - Sidekiq enqueues `Geo::EventWorker` jobs with arguments like "Job Artifact with ID 123 was updated" for each item in the queue, and tracks the enqueued Sidekiq job IDs. + - Sidekiq stops enqueuing `Geo::EventWorker` jobs when "maximum concurrency limit" settings are reached + - Sidekiq loops doing this kind of work until it has no more to do +- Sidekiq picks up `Geo::EventWorker` job + - Sidekiq marks the `job_artifact_registry` row as "started sync" + - Sidekiq does a GET request on an API endpoint at the primary Geo site and downloads the file + - Sidekiq marks the `job_artifact_registry` row as "synced" and "pending verification" + +##### Verifying a new job artifact + +Primary site: + +```mermaid +sequenceDiagram + participant Ru as Runner + participant P as Puma + participant DB as PostgreSQL + participant SC as Sidekiq-cron + participant Rd as Redis + participant S as Sidekiq + participant F as Filesystem + Ru->>P: Upload artifact + P->>DB: Insert `ci_job_artifacts` + P->>DB: Insert `ci_job_artifact_states` + SC->>Rd: Enqueue `Geo::VerificationCronWorker` + S->>Rd: Pick up `Geo::VerificationCronWorker` + S->>DB: Query `ci_job_artifact_states` + S->>Rd: Enqueue `Geo::VerificationBatchWorker` + S->>Rd: Pick up `Geo::VerificationBatchWorker` + S->>DB: Query `ci_job_artifact_states` + S->>DB: Update `ci_job_artifact_states` row, "started" + S->>F: Checksum file + S->>DB: Update `ci_job_artifact_states` row, "succeeded" +``` + +- A [Runner](https://docs.gitlab.com/runner/) uploads an artifact +- [Puma](architecture.md#puma) creates a `ci_job_artifacts` row +- Puma creates a `ci_job_artifact_states` row to store verification state. + - The row is marked "pending verification" +- [Sidekiq-cron](https://github.com/ondrejbartas/sidekiq-cron) enqueues a `Geo::VerificationCronWorker` job every minute +- [Sidekiq](architecture.md#sidekiq) picks up the `Geo::VerificationCronWorker` job + - Sidekiq queries `ci_job_artifact_states` for the number of rows marked "pending verification" or "failed verification and ready to retry" + - Sidekiq enqueues one or more `Geo::VerificationBatchWorker` jobs, limited by the "maximum verification concurrency" setting +- Sidekiq picks up `Geo::VerificationBatchWorker` job + - Sidekiq queries `ci_job_artifact_states` for rows marked "pending verification" + - If the previous step yielded less than 10 rows, then Sidekiq queries `ci_job_artifact_states` for rows marked "failed verification and ready to retry" + - For each row + - Sidekiq marks it "started verification" + - Sidekiq gets the SHA256 checksum of the file + - Sidekiq saves the checksum in the row and marks it "succeeded verification" + - Now secondary Geo sites can compare against this checksum + +Secondary site: + +```mermaid +sequenceDiagram + participant SC as Sidekiq-cron + participant R as Redis + participant S as Sidekiq + participant TDB as PostgreSQL Tracking DB + participant F as Filesystem + participant DB as PostgreSQL + SC->>R: Enqueue `Geo::VerificationCronWorker` + S->>R: Pick up `Geo::VerificationCronWorker` + S->>TDB: Query `job_artifact_registry` + S->>R: Enqueue `Geo::VerificationBatchWorker` + S->>R: Pick up `Geo::VerificationBatchWorker` + S->>TDB: Query `job_artifact_registry` + S->>TDB: Update `job_artifact_registry` row, "started" + S->>F: Checksum file + S->>DB: Query `ci_job_artifact_states` + S->>TDB: Update `job_artifact_registry` row, "succeeded" +``` + +- After the artifact is successfully synced, it becomes "pending verification" +- [Sidekiq-cron](https://github.com/ondrejbartas/sidekiq-cron) enqueues a `Geo::VerificationCronWorker` job every minute +- [Sidekiq](architecture.md#sidekiq) picks up the `Geo::VerificationCronWorker` job + - Sidekiq queries `job_artifact_registry` in the [PostgreSQL Geo Tracking Database](#tracking-database) for the number of rows marked "pending verification" or "failed verification and ready to retry" + - Sidekiq enqueues one or more `Geo::VerificationBatchWorker` jobs, limited by the "maximum verification concurrency" setting +- Sidekiq picks up `Geo::VerificationBatchWorker` job + - Sidekiq queries `job_artifact_registry` in the PostgreSQL Geo Tracking Databasef for rows marked "pending verification" + - If the previous step yielded less than 10 rows, then Sidekiq queries `job_artifact_registry` for rows marked "failed verification and ready to retry" + - For each row + - Sidekiq marks it "started verification" + - Sidekiq gets the SHA256 checksum of the file + - Sidekiq saves the checksum in the row + - Sidekiq compares the checksum against the checksum in the `ci_job_artifact_states` row which was replicated by PostgreSQL + - If the checksum matches, then Sidekiq marks the `job_artifact_registry` row "succeeded verification" ## Authentication @@ -241,6 +440,22 @@ ignores items in object storage. Either: ## Verification +### Verification states + +The following diagram illustrates how the verification works. Some allowed transitions are omitted for clarity. + +```mermaid +stateDiagram-v2 + Pending --> Started + Pending --> Disabled: No primary checksum + Disabled --> Started: Primary checksum succeeded + Started --> Succeeded + Started --> Failed + Succeeded --> Pending: Mark for reverify + Failed --> Pending: Mark for reverify + Failed --> Started: Retry +``` + ### Repository verification Repositories are verified with a checksum. @@ -252,7 +467,12 @@ basically hashes all Git refs together and stores that hash in the The **secondary** site does the same to calculate the hash of its clone, and compares the hash with the value the **primary** site calculated. If there is a mismatch, Geo will mark this as a mismatch -and the administrator can see this in the [Geo Admin Area](../user/admin_area/geo_nodes.md). +and the administrator can see this in the [Geo Admin Area](../user/admin_area/geo_sites.md). + +## Geo proxying + +Geo secondaries can proxy web requests to the primary. +Read more on the [Geo proxying (development) page](geo/proxying.md). ## Glossary @@ -303,10 +523,7 @@ events include: - Job Artifact Deleted event - Upload Deleted event -### Geo Log Cursor - -The process running on the **secondary** site that looks for new -`Geo::EventLog` rows. +See [Geo Log Cursor daemon](#geo-log-cursor-daemon). ## Code features @@ -415,7 +632,7 @@ We switch and filter from each event by the `event_name` field. ### Geo Log Cursor (GitLab 10.0 and up) In GitLab 10.0 and later, [System Webhooks](#system-hooks-gitlab-87-to-95) are no longer -used and Geo Log Cursor is used instead. The Log Cursor traverses the +used and [Geo Log Cursor](#geo-log-cursor-daemon) is used instead. The Log Cursor traverses the `Geo::EventLog` rows to see if there are changes since the last time the log was checked and will handle repository updates, deletes, changes, and renames. diff --git a/doc/development/geo/proxying.md b/doc/development/geo/proxying.md new file mode 100644 index 00000000000..41c7f426c6f --- /dev/null +++ b/doc/development/geo/proxying.md @@ -0,0 +1,356 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Geo proxying + +With Geo proxying, secondaries now proxy web requests through Workhorse to the primary, so users navigating to the +secondary see a read-write UI, and are able to do all operations that they can do on the primary. + +## Request life cycle + +### Top-level view + +The proxying interaction can be explained at a high level through the following diagram: + +```mermaid +sequenceDiagram +actor client +participant secondary +participant primary + +client->>secondary: GET /explore +secondary-->>primary: GET /explore (proxied) +primary-->>secondary: HTTP/1.1 200 OK [..] +secondary->>client: HTTP/1.1 200 OK [..] +``` + +### Proxy detection mechanism + +To know whether or not it should proxy requests to the primary, and the URL of the primary (as it is stored in +the database), Workhorse polls the internal API when Geo is enabled. When proxying should be enabled, the internal +API responds with the primary URL and JWT-signed data that is passed on to the primary for every request. + +```mermaid +sequenceDiagram + participant W as Workhorse (secondary) + participant API as Internal Rails API + W->API: GET /api/v4/geo/proxy (internal) + loop Poll every 10 seconds + API-->W: {geo_proxy_primary_url, geo_proxy_extra_data}, update config + end +``` + +### In-depth request flow and local data acceleration compared with proxying + +Detailing implementation, Workhorse on the secondary (requested) site decides whether to proxy the data or not. If it +can "accelerate" the data type (that is, can serve locally to save a roundtrip request), it returns the data +immediately. Otherwise, traffic is sent to the primary's internal URL, served by Workhorse on the primary exactly +as a direct request would. The response is then be proxied back to the user through the secondary Workhorse in the +same connection. + +```mermaid +flowchart LR + A[Client]--->W1["Workhorse (secondary)"] + W1 --> W1C[Serve data locally?] + W1C -- "Yes" ----> W1 + W1C -- "No (proxy)" ----> W2["Workhorse (primary)"] + W2 --> W1 ----> A +``` + +## Sign-in + +### Requests proxied to the primary requiring authorization + +```mermaid +sequenceDiagram +autoNumber +participant Client +participant Secondary +participant Primary + +Client->>Secondary: `/group/project` request +Secondary->>Primary: proxy /group/project +opt primary not signed in +Primary-->>Secondary: 302 redirect +Secondary-->>Client: proxy 302 redirect +Client->>Secondary: /users/sign_in +Secondary->>Primary: proxy /users/sign_in +Note right of Primary: authentication happens, POST to same URL etc +Primary-->>Secondary: 302 redirect +Secondary-->>Client: proxy 302 redirect +Client->>Secondary: /group/project +Secondary->>Primary: proxy /group/project +end +Primary-->>Secondary: /group/project logged in response (session on primary created) +Secondary-->>Client: proxy full response +``` + +### Requests requiring a user session on the secondary + +At the moment, this flow only applies to Project Replication Details and Design Replication Details in the Geo Admin +Area. For more context, see +[View replication data on the primary site](../../administration/geo/index.md#view-replication-data-on-the-primary-site). + +```mermaid +sequenceDiagram +autoNumber +participant Client +participant Secondary +participant Primary + +Client->>Secondary: `admin/geo/replication/projects` request +opt secondary not signed in +Secondary-->>Client: 302 redirect +Client->>Secondary: /users/auth/geo/sign_in +Secondary-->>Client: 302 redirect +Client->>Secondary: /oauth/geo/auth/geo/sign_in +Secondary-->>Client: 302 redirect +Client->>Secondary: /oauth/authorize +Secondary->>Primary: proxy /oauth/authorize +opt primary not signed in +Primary-->>Secondary: 302 redirect +Secondary-->>Client: proxy 302 redirect +Client->>Secondary: /users/sign_in +Secondary->>Primary: proxy /users/sign_in +Note right of Primary: authentication happens, POST to same URL etc +end +Primary-->>Secondary: 302 redirect +Secondary-->>Client: proxy 302 redirect +Client->>Secondary: /oauth/geo/callback +Secondary-->>Client: 302 redirect +Client->>Secondary: admin/geo/replication/projects +end +Secondary-->>Client: admin/geo/replication/projects logged in response (session on both primary and secondary) +``` + +## Git pull + +For historical reasons, the `push_from_secondary` path is used to forward a Git pull. There is [an issue proposing to +rename this route](https://gitlab.com/gitlab-org/gitlab/-/issues/292690) to avoid confusion. + +### Git pull over HTTP(s) + +#### Accelerated repositories + +When a repository exists on the secondary and we detect is up to date with the primary, we serve it directly instead of +proxying. + +```mermaid +sequenceDiagram +participant C as Git client +participant Wsec as "Workhorse (secondary)" +participant Rsec as "Rails (secondary)" +participant Gsec as "Gitaly (secondary)" +C->>Wsec: GET /foo/bar.git/info/refs/?service=git-upload-pack +Wsec->>Rsec: <internal API check> +note over Rsec: decide that the repo is synced and up to date +Rsec-->>Wsec: 401 Unauthorized +Wsec-->>C: <response> +C->>Wsec: GET /foo/bar.git/info/refs/?service=git-upload-pack +Wsec->>Rsec: <internal API check> +Rsec-->>Wsec: Render Workhorse OK +Wsec-->>C: 200 OK +C->>Wsec: POST /foo/bar.git/git-upload-pack +Wsec->>Rsec: GitHttpController#git_receive_pack +Rsec-->>Wsec: Render Workhorse OK +Wsec->>Gsec: Workhorse gets the connection details from Rails, connects to Gitaly: SmartHTTP Service, UploadPack RPC (check the proto for details) +Gsec-->>Wsec: Return a stream of Proto messages +Wsec-->>C: Pipe messages to the Git client +``` + +#### Proxied repositories + +If a requested repository isn't synced, or we detect is not up to date, the request will be proxied to the primary, in +order to get the latest version of the changes. + +```mermaid +sequenceDiagram +participant C as Git client +participant Wsec as "Workhorse (secondary)" +participant Rsec as "Rails (secondary)" +participant W as "Workhorse (primary)" +participant R as "Rails (primary)" +participant G as "Gitaly (primary)" +C->>Wsec: GET /foo/bar.git/info/refs/?service=git-upload-pack +Wsec->>Rsec: <response> +note over Rsec: decide that the repo is out of date +Rsec-->>Wsec: 302 Redirect to /-/push_from_secondary/2/foo/bar.git/info/refs?service=git-upload-pack +Wsec-->>C: <response> +C->>Wsec: GET /-/push_from_secondary/2/foo/bar.git/info/refs/?service=git-upload-pack +Wsec->>W: <proxied request> +W->>R: <data> +R-->>W: 401 Unauthorized +W-->>Wsec: <proxied response> +Wsec-->>C: <response> +C->>Wsec: GET /-/push_from_secondary/2/foo/bar.git/info/refs/?service=git-upload-pack +note over W: proxied +Wsec->>W: <proxied request> +W->>R: <data> +R-->>W: Render Workhorse OK +W-->>Wsec: <proxied response> +Wsec-->>C: <response> +C->>Wsec: POST /-/push_from_secondary/2/foo/bar.git/git-upload-pack +Wsec->>W: <proxied request> +W->>R: GitHttpController#git_receive_pack +R-->>W: Render Workhorse OK +W->>G: Workhorse gets the connection details from Rails, connects to Gitaly: SmartHTTP Service, UploadPack RPC (check the proto for details) +G-->>W: Return a stream of Proto messages +W-->>Wsec: Pipe messages to the Git client +Wsec-->>C: Return piped messages from Git +``` + +### Git pull over SSH + +As SSH operations go through GitLab Shell instead of Workhorse, they are not proxied through the mechanism used for +Workhorse requests. With SSH operations, they are proxied as Git HTTP requests to the primary site by the secondary +Rails internal API. + +#### Accelerated repositories + +When a repository exists on the secondary and we detect is up to date with the primary, we serve it directly instead of +proxying. + +```mermaid +sequenceDiagram +participant C as Git client +participant S as GitLab Shell (secondary) +participant I as Internal API (secondary Rails) +participant G as Gitaly (secondary) +C->>S: git pull +S->>I: SSH key validation (api/v4/internal/authorized_keys?key=..) +I-->>S: HTTP/1.1 200 OK +S->>G: InfoRefs:UploadPack RPC +G-->>S: stream Git response back +S-->>C: stream Git response back +C-->>S: stream Git data to push +S->>G: UploadPack RPC +G-->>S: stream Git response back +S-->>C: stream Git response back +``` + +#### Proxied repositories + +If a requested repository isn't synced, or we detect is not up to date, the request will be proxied to the primary, in +order to get the latest version of the changes. + +```mermaid +sequenceDiagram +participant C as Git client +participant S as GitLab Shell (secondary) +participant I as Internal API (secondary Rails) +participant P as Primary API +C->>S: git pull +S->>I: SSH key validation (api/v4/internal/authorized_keys?key=..) +I-->>S: HTTP/1.1 300 (custom action status) with {endpoint, msg, primary_repo} +S->>I: POST /api/v4/geo/proxy_git_ssh/info_refs_upload_pack +I->>P: POST $PRIMARY/foo/bar.git/info/refs/?service=git-upload-pack +P-->>I: HTTP/1.1 200 OK +I-->>S: <response> +S-->>C: return Git response from primary +C-->>S: stream Git data to push +S->>I: POST /api/v4/geo/proxy_git_ssh/upload_pack +I->>P: POST $PRIMARY/foo/bar.git/git-upload-pack +P-->>I: HTTP/1.1 200 OK +I-->>S: <response> +S-->>C: return Git response from primary +``` + +## Git push + +### Unified URLs + +With unified URLs, a push will redirect to a local path formatted as `/-/push_from_secondary/$SECONDARY_ID/*`. Further +requests through this path will be proxied to the primary, which will handle the push. + +#### Git push over SSH + +As SSH operations go through GitLab Shell instead of Workhorse, they are not proxied through the mechanism used for +Workhorse requests. With SSH operations, they are proxied as Git HTTP requests to the primary site by the secondary +Rails internal API. + +```mermaid +sequenceDiagram +participant C as Git client +participant S as GitLab Shell (secondary) +participant I as Internal API (secondary Rails) +participant P as Primary API +C->>S: git push +S->>I: SSH key validation (api/v4/internal/authorized_keys?key=..) +I-->>S: HTTP/1.1 300 (custom action status) with {endpoint, msg, primary_repo} +S->>I: POST /api/v4/geo/proxy_git_ssh/info_refs_receive_pack +I->>P: POST $PRIMARY/foo/bar.git/info/refs/?service=git-receive-pack +P-->>I: HTTP/1.1 200 OK +I-->>S: <response> +S-->>C: return Git response from primary +C-->>S: stream Git data to push +S->>I: POST /api/v4/geo/proxy_git_ssh/receive_pack +I->>P: POST $PRIMARY/foo/bar.git/git-receive-pack +P-->>I: HTTP/1.1 200 OK +I-->>S: <response> +S-->>C: return Git response from primary +``` + +#### Git push over HTTP(s) + +```mermaid +sequenceDiagram +participant C as Git client +participant Wsec as Workhorse (secondary) +participant W as Workhorse (primary) +participant R as Rails (primary) +participant G as Gitaly (primary) +C->>Wsec: GET /foo/bar.git/info/refs/?service=git-receive-pack +Wsec->>C: 302 Redirect to /-/push_from_secondary/2/foo/bar.git/info/refs?service=git-receive-pack +C->>Wsec: GET /-/push_from_secondary/2/foo/bar.git/info/refs/?service=git-receive-pack +Wsec->>W: <proxied request> +W->>R: <data> +R-->>W: 401 Unauthorized +W-->>Wsec: <proxied response> +Wsec-->>C: <response> +C->>Wsec: GET /-/push_from_secondary/2/foo/bar.git/info/refs/?service=git-receive-pack +Wsec->>W: <proxied request> +W->>R: <data> +R-->>W: Render Workhorse OK +W-->>Wsec: <proxied response> +Wsec-->>C: <response> +C->>Wsec: POST /-/push_from_secondary/2/foo/bar.git/git-receive-pack +Wsec->>W: <proxied request> +W->>R: GitHttpController:git_receive_pack +R-->>W: Render Workhorse OK +W->>G: Get connection details from Rails and connects to SmartHTTP Service, ReceivePack RPC +G-->>W: Return a stream of Proto messages +W-->>Wsec: Pipe messages to the Git client +Wsec-->>C: Return piped messages from Git +``` + +### Git push over HTTP with Separate URLs + +With separate URLs, the secondary will redirect to a URL formatted like `$PRIMARY/-/push_from_secondary/$SECONDARY_ID/*`. + +```mermaid +sequenceDiagram +participant Wsec as Workhorse (secondary) +participant C as Git client +participant W as Workhorse (primary) +participant R as Rails (primary) +participant G as Gitaly (primary) +C->>Wsec: GET $SECONDARY/foo/bar.git/info/refs/?service=git-receive-pack +Wsec->>C: 302 Redirect to $PRIMARY/-/push_from_secondary/2/foo/bar.git/info/refs?service=git-receive-pack +C->>W: GET $PRIMARY/-/push_from_secondary/2/foo/bar.git/info/refs/?service=git-receive-pack +W->>R: <data> +R-->>W: 401 Unauthorized +W-->>C: <response> +C->>W: GET /-/push_from_secondary/2/foo/bar.git/info/refs/?service=git-receive-pack +W->>R: <data> +R-->>W: Render Workhorse OK +W-->>C: <response> +C->>W: POST /-/push_from_secondary/2/foo/bar.git/git-receive-pack +W->>R: GitHttpController:git_receive_pack +R-->>W: Render Workhorse OK +W->>G: Get connection details from Rails and connects to SmartHTTP Service, ReceivePack RPC +G-->>W: Return a stream of Proto messages +W-->>C: Pipe messages to the Git client +``` diff --git a/doc/development/git_object_deduplication.md b/doc/development/git_object_deduplication.md index 3ac24b19fc2..1a864ef81f0 100644 --- a/doc/development/git_object_deduplication.md +++ b/doc/development/git_object_deduplication.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # How Git object deduplication works in GitLab diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index 0743a03ddac..8a0cf8e7717 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Gitaly developers guide @@ -388,3 +387,28 @@ When you are using GDK, you can set it up with: 1. Start the database: `gdk start db` 1. Load the environment from GDK: `eval $(cd ../gitaly && gdk env)` 1. Create the database: `createdb --encoding=UTF8 --locale=C --echo praefect_test` + +## Git references used by Gitaly + +Gitaly uses many Git references ([refs](https://git-scm.com/docs/gitglossary#Documentation/gitglossary.txt-aiddefrefaref)) to provide Git services to GitLab. + +### Standard Git references + +These standard Git references are used by GitLab (through Gitaly) in any Git repository: + +- `refs/heads/`. Used for branches. See the [`git branch`](https://git-scm.com/docs/git-branch) documentation. +- `refs/tags/`. Used for tags. See the [`git tag`](https://git-scm.com/docs/git-tag) documentation. + +### GitLab-specific references + +These GitLab-specific references are used exclusively by GitLab (through Gitaly): + +- `refs/keep-around/<object-id>`. References to commits that have pipeline jobs or merge requests. The `object-id` points to the commit the pipeline was run on. +- `refs/merge-requests/<merge-request-iid>/`. [Merges](https://git-scm.com/docs/git-merge) merge two histories together. This ref namespace tracks information about a + merge using the following refs under it: + - `head`. Current `HEAD` of the merge request. + - `merge`. Commit for the merge request. Every merge request creates a commit object under `refs/keep-around`. + - If [merge trains are enabled](../ci/pipelines/merge_trains.md): `train`. Commit for the merge train. +- `refs/pipelines/<pipeline-iid>`. References to pipelines. Temporarily used to store the pipeline commit object ID. +- `refs/environments/<environment-slug>`. References to commits where deployments to environments were performed. +- `refs/heads/revert-<source-commit-short-object-id>`. References to the commit's object ID created when [reverting changes](../user/project/merge_requests/revert_changes.md). diff --git a/doc/development/gitlab_flavored_markdown/specification_guide/index.md b/doc/development/gitlab_flavored_markdown/specification_guide/index.md index cedf44cf1fc..80837506037 100644 --- a/doc/development/gitlab_flavored_markdown/specification_guide/index.md +++ b/doc/development/gitlab_flavored_markdown/specification_guide/index.md @@ -69,11 +69,11 @@ serve as input to automated conformance tests. It is > This document attempts to specify Markdown syntax unambiguously. It contains many > examples with side-by-side Markdown and HTML. These examples are intended to double as conformance tests. -The HTML-rendered versions of the specifications: +Here are the HTML-rendered versions of the specifications: - [GitLab Flavored Markdown (GLFM) specification](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/output/spec.html), which extends the: -- [GitHub Flavored Markdown (GFM) specification](https://github.github.com/gfm/), which extends the: -- [CommonMark specification](https://spec.commonmark.org/0.30/) +- [GitHub Flavored Markdown (GFM) specification](https://github.github.com/gfm/) (rendered from the [source `spec.txt` for GFM specification](https://github.com/github/cmark-gfm/blob/master/test/spec.txt)), which extends the: +- [CommonMark specification](https://spec.commonmark.org/0.30/) (rendered from the [source `spec.txt` for CommonMark specification](https://github.com/commonmark/commonmark-spec/blob/master/spec.txt)) NOTE: The creation of the @@ -136,10 +136,10 @@ NOTE: #### Markdown snapshot testing _Markdown snapshot testing_ refers to the automated testing performed in -the GitLab codebase, which is driven by snapshot fixture data derived from the -GLFM specification. It consists of both backend RSpec tests and frontend Jest tests -which use the fixture data. This fixture data is contained in YAML files. These files -can be generated and updated based on the Markdown examples in the specification, +the GitLab codebase, which is driven by "example_snapshots" fixture data derived from all of +the examples in the GLFM specification. It consists of both backend RSpec tests and frontend Jest +tests which use the fixture data. This fixture data is contained in YAML files. These files +are generated and updated based on the Markdown examples in the specification, and the existing GLFM parser and render implementations. They may also be manually updated as necessary to test-drive incomplete implementations. Regarding the terminology used here: @@ -159,7 +159,7 @@ Regarding the terminology used here: [Jest snapshot testing](https://jestjs.io/docs/snapshot-testing), as used elsewhere in the GitLab frontend testing suite. However, the Markdown snapshot testing does follow the same philosophy and patterns as Jest snapshot testing: - 1. Snapshot fixture data is represented as files which are checked into source control. + 1. Snapshot example fixture data is represented as files which are checked into source control. 1. The files can be automatically generated and updated based on the implementation of the code under tests. 1. The files can also be manually updated when necessary, for example, to test-drive @@ -168,9 +168,15 @@ Regarding the terminology used here: [Rails database fixture files](https://api.rubyonrails.org/classes/ActiveRecord/FixtureSet.html). It instead refers to _test fixtures_ in the [more generic definition](https://en.wikipedia.org/wiki/Test_fixture#Software), - as input data to support automated testing. However, fixture files still exist, so - they are colocated under the `spec/fixtures` directory with the rest of - the fixture data for the GitLab Rails application. + as input data to support automated testing. +1. These example snapshots fixture files are generated from and closely related to the rest of the + GLFM specification. Therefore, the `example_snapshots` directory is colocated under the + `glfm_specification` directory with the rest of the + GLFM [specification files](#specification-files). They are intentionally _not_ + located under the `spec/fixtures` directory with the rest of + the fixture data for the GitLab Rails application. In practice, developers have found + it simpler and more understandable to have everything under the `glfm_specification` directory + rather than splitting these files into the `spec/fixtures` directory. <!-- vale gitlab.InclusionCultural = YES --> @@ -395,9 +401,10 @@ The documentation on the implementation is split into three sections: 1. [Scripts](#scripts). 1. [Specification files](#specification-files). -1. Example snapshot files: These YAML files are used as input data +1. [Example snapshot files](#example-snapshot-files): + These YAML files are used as input data or fixtures to drive the various tests, and are located under - `spec/fixtures/glfm/example_snapshots`. All example snapshot files are automatically + `glfm_specification/example_snapshots`. All example snapshot files are automatically generated based on the specification files and the implementation of the parsers and renderers. However, they can also be directly edited if necessary, such as to test-drive an incomplete implementation. @@ -662,16 +669,16 @@ controls the behavior of the [scripts](#scripts) and [tests](#types-of-markdown- The following optional entries are supported for each example. They all default to `false`: - `skip_update_example_snapshots`: When true, skips any addition or update of any this example's entries - in the [`spec/fixtures/glfm/example_snapshots/html.yml`](#specfixturesglfmexample_snapshotshtmlyml) file - or the [`spec/fixtures/glfm/example_snapshots/prosemirror_json.yml`](#specfixturesglfmexample_snapshotsprosemirror_jsonyml) file. + in the [`glfm_specification/example_snapshots/html.yml`](#glfm_specificationexample_snapshotshtmlyml) file + or the [`glfm_specification/example_snapshots/prosemirror_json.yml`](#glfm_specificationexample_snapshotsprosemirror_jsonyml) file. If this value is truthy, then no other `skip_update_example_snapshot_*` entries can be truthy, and an error is raised if any of them are. - `skip_update_example_snapshot_html_static`: When true, skips addition or update of this example's [static HTML](#static-html) - entry in the [`spec/fixtures/glfm/example_snapshots/html.yml`](#specfixturesglfmexample_snapshotshtmlyml) file. + entry in the [`glfm_specification/example_snapshots/html.yml`](#glfm_specificationexample_snapshotshtmlyml) file. - `skip_update_example_snapshot_html_wysiwyg`: When true, skips addition or update of this example's [WYSIWYG HTML](#wysiwyg-html) - entry in the [`spec/fixtures/glfm/example_snapshots/html.yml`](#specfixturesglfmexample_snapshotshtmlyml) file. + entry in the [`glfm_specification/example_snapshots/html.yml`](#glfm_specificationexample_snapshotshtmlyml) file. - `skip_update_example_snapshot_prosemirror_json`: When true, skips addition or update of this example's - entry in the [`spec/fixtures/glfm/example_snapshots/prosemirror_json.yml`](#specfixturesglfmexample_snapshotsprosemirror_jsonyml) file. + entry in the [`glfm_specification/example_snapshots/prosemirror_json.yml`](#glfm_specificationexample_snapshotsprosemirror_jsonyml) file. - `skip_running_conformance_static_tests`: When true, skips running the [Markdown conformance tests](#markdown-conformance-testing) of the [static HTML](#static-html) for this example. - `skip_running_conformance_wysiwyg_tests`: When true, skips running the [Markdown conformance tests](#markdown-conformance-testing) @@ -681,7 +688,7 @@ The following optional entries are supported for each example. They all default - `skip_running_snapshot_wysiwyg_html_tests`: When true, skips running the [Markdown snapshot tests](#markdown-snapshot-testing) of the [WYSIWYG HTML](#wysiwyg-html) for this example. - `skip_running_snapshot_prosemirror_json_tests`: When true, skips running the [Markdown snapshot tests](#markdown-snapshot-testing) - of the [ProseMirror JSON](#specfixturesglfmexample_snapshotsprosemirror_jsonyml) for this example. + of the [ProseMirror JSON](#glfm_specificationexample_snapshotsprosemirror_jsonyml) for this example. `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.yml` sample entry: @@ -808,9 +815,9 @@ key in `glfm_specification/input/gitlab_flavored_markdown/glfm_example_status.ym can be used to disable automatic generation of some examples. They can instead be manually edited as necessary to help drive the implementations. -#### `spec/fixtures/glfm/example_snapshots/examples_index.yml` +#### `glfm_specification/example_snapshots/examples_index.yml` -[`spec/fixtures/glfm/example_snapshots/examples_index.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/fixtures/glfm/example_snapshots/examples_index.yml) +[`glfm_specification/example_snapshots/examples_index.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/example_snapshots/examples_index.yml) is the main list of all CommonMark, GFM, and GLFM example names, each with a unique canonical name. @@ -836,7 +843,7 @@ CommonMark, GFM, and GLFM example names, each with a unique canonical name. examples where multiple examples exist for the same Section 7 subsection are added to the end of the sub-section. -`spec/fixtures/glfm/example_snapshots/examples_index.yml` sample entries: +`glfm_specification/example_snapshots/examples_index.yml` sample entries: ```yaml 02_01_preliminaries_characters_and_lines_1: @@ -856,10 +863,10 @@ CommonMark, GFM, and GLFM example names, each with a unique canonical name. source_specification: gitlab ``` -#### `spec/fixtures/glfm/example_snapshots/markdown.yml` +#### `glfm_specification/example_snapshots/markdown.yml` -[`spec/fixtures/glfm/example_snapshots/markdown.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/fixtures/glfm/example_snapshots/markdown.yml) contains the original Markdown -for each entry in `spec/fixtures/glfm/example_snapshots/examples_index.yml` +[`glfm_specification/example_snapshots/markdown.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/example_snapshots/markdown.yml) contains the original Markdown +for each entry in `glfm_specification/example_snapshots/examples_index.yml` - For CommonMark and GFM Markdown, it is generated (or updated) from the standard GFM @@ -868,17 +875,17 @@ for each entry in `spec/fixtures/glfm/example_snapshots/examples_index.yml` `glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt` input specification file. -`spec/fixtures/glfm/example_snapshots/markdown.yml` sample entry: +`glfm_specification/example_snapshots/markdown.yml` sample entry: ```yaml 06_04_inlines_emphasis_and_strong_emphasis_1: | *foo bar* ``` -#### `spec/fixtures/glfm/example_snapshots/html.yml` +#### `glfm_specification/example_snapshots/html.yml` -[`spec/fixtures/glfm/example_snapshots/html.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/fixtures/glfm/example_snapshots/html.yml) -contains the HTML for each entry in `spec/fixtures/glfm/example_snapshots/examples_index.yml` +[`glfm_specification/example_snapshots/html.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/example_snapshots/html.yml) +contains the HTML for each entry in `glfm_specification/example_snapshots/examples_index.yml` Three types of entries exist, with different HTML for each: @@ -889,13 +896,13 @@ Three types of entries exist, with different HTML for each: `glfm_specification/input/gitlab_flavored_markdown/glfm_canonical_examples.txt`. - **Static** - This is the static (backend (Ruby)-generated) HTML for each entry in - `spec/fixtures/glfm/example_snapshots/examples_index.yml`. + `glfm_specification/example_snapshots/examples_index.yml`. - It is generated/updated from backend [Markdown API](../../../api/markdown.md) (or the underlying internal classes) via the `update-example-snapshots.rb` script, but can be manually updated for static examples with incomplete implementations. - **WYSIWYG** - The WYSIWYG (frontend, JavaScript-generated) HTML for each entry in - `spec/fixtures/glfm/example_snapshots/examples_index.yml`. + `glfm_specification/example_snapshots/examples_index.yml`. - It is generated (or updated) from the frontend Content Editor implementation via the `update-example-snapshots.rb` script. It can be manually updated for WYSIWYG examples with incomplete implementations. @@ -903,7 +910,7 @@ Three types of entries exist, with different HTML for each: Any exceptions or failures which occur when generating HTML are replaced with an `Error - check implementation` value. -`spec/fixtures/glfm/example_snapshots/html.yml` sample entry: +`glfm_specification/example_snapshots/html.yml` sample entry: ```yaml 06_04_inlines_emphasis_and_strong_emphasis_1: @@ -919,16 +926,16 @@ NOTE: The actual `static` or `WYSIWYG` entries may differ from the example `html.yml`, depending on how the implementations evolve. -#### `spec/fixtures/glfm/example_snapshots/prosemirror_json.yml` +#### `glfm_specification/example_snapshots/prosemirror_json.yml` -[`spec/fixtures/glfm/example_snapshots/prosemirror_json.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/fixtures/glfm/example_snapshots/prosemirror_json.yml) -contains the ProseMirror JSON for each entry in `spec/fixtures/glfm/example_snapshots/examples_index.yml` +[`glfm_specification/example_snapshots/prosemirror_json.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/glfm_specification/example_snapshots/prosemirror_json.yml) +contains the ProseMirror JSON for each entry in `glfm_specification/example_snapshots/examples_index.yml` - It is generated (or updated) from the frontend code via the `update-example-snapshots.rb` script, but can be manually updated for examples with incomplete implementations. - Any exceptions or failures when generating are replaced with a `Error - check implementation` value. -`spec/fixtures/glfm/example_snapshots/prosemirror_json.yml` sample entry: +`glfm_specification/example_snapshots/prosemirror_json.yml` sample entry: ```yaml 06_04_inlines_emphasis_and_strong_emphasis_1: |- diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index f5b0da2f162..1a11321b70f 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -443,6 +443,43 @@ of the Code Review Comments page on the Go wiki for more details. Most editors/IDEs allow you to run commands before/after saving a file, you can set it up to run `goimports -local gitlab.com/gitlab-org` so that it's applied to every file when saving. +### Initializing slices + +If initializing a slice, provide a capacity where possible to avoid extra +allocations. + +<table> +<tr><th>:white_check_mark: Do</th><th>:x: Don't</th></tr> +<tr> + <td> + + ```golang + s2 := make([]string, 0, size) + for _, val := range s1 { + s2 = append(s2, val) + } + ``` + + </td> + <td> + + ```golang + var s2 []string + for _, val := range s1 { + s2 = append(s2, val) + } + ``` + + </td> +</tr> +</table> + +If no capacity is passed to `make` when creating a new slice, `append` +will continuously resize the slice's backing array if it cannot hold +the values. Providing the capacity ensures that allocations are kept +to a minimum. It is recommended that the [`prealloc`](https://github.com/alexkohler/prealloc) +golanci-lint rule automatically check for this. + ### Analyzer Tests The conventional Secure [analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/) has a [`convert` function](https://gitlab.com/gitlab-org/security-products/analyzers/command/-/blob/main/convert.go#L15-17) that converts SAST/DAST scanner reports into [GitLab Security Reports](https://gitlab.com/gitlab-org/security-products/security-report-schemas). When writing tests for the `convert` function, we should make use of [test fixtures](https://dave.cheney.net/2016/05/10/test-fixtures-in-go) using a `testdata` directory at the root of the analyzer's repository. The `testdata` directory should contain two subdirectories: `expect` and `reports`. The `reports` directory should contain sample SAST/DAST scanner reports which are passed into the `convert` function during the test setup. The `expect` directory should contain the expected GitLab Security Report that the `convert` returns. See Secret Detection for an [example](https://gitlab.com/gitlab-org/security-products/analyzers/secrets/-/blob/160424589ef1eed7b91b59484e019095bc7233bd/convert_test.go#L13-66). diff --git a/doc/development/i18n/externalization.md b/doc/development/i18n/externalization.md index 2aea15de443..18704fc2b60 100644 --- a/doc/development/i18n/externalization.md +++ b/doc/development/i18n/externalization.md @@ -411,6 +411,56 @@ use `%{created_at}` in Ruby but `%{createdAt}` in JavaScript. Make sure to // => When x == 2: 'Last 2 days' ``` +- In Vue: + + One of [the recommended ways to organize translated strings for Vue files](#vue-files) is to extract them into a `constants.js` file. + That can be difficult to do when there are pluralized strings because the `count` variable won't be known inside the constants file. + To overcome this, we recommend creating a function which takes a `count` argument: + + ```javascript + // .../feature/constants.js + import { n__ } from '~/locale'; + + export const I18N = { + // Strings that are only singular don't need to be a function + someDaysRemain: __('Some days remain'), + daysRemaining(count) { return n__('%d day remaining', '%d days remaining', count); }, + }; + ``` + + Then within a Vue component the function can be used to retrieve the correct pluralization form of the string: + + ```javascript + // .../feature/components/days_remaining.vue + import { sprintf } from '~/locale'; + import { I18N } from '../constants'; + + <script> + export default { + props: { + days: { + type: Number, + required: true, + }, + }, + i18n: I18N, + }; + </script> + + <template> + <div> + <span> + A singular string: + {{ $options.i18n.someDaysRemain }} + </span> + <span> + A plural string: + {{ $options.i18n.daysRemaining(days) }} + </span> + </div> + </template> + ``` + The `n_` and `n__` methods should only be used to fetch pluralized translations of the same string, not to control the logic of showing different strings for different quantities. Some languages have different quantities of target plural forms. diff --git a/doc/development/i18n/proofreader.md b/doc/development/i18n/proofreader.md index 8231cf4316b..cee078ca891 100644 --- a/doc/development/i18n/proofreader.md +++ b/doc/development/i18n/proofreader.md @@ -46,6 +46,8 @@ are very appreciative of the work done by translators and proofreaders! - scootergrisen - [GitLab](https://gitlab.com/scootergrisen), [Crowdin](https://crowdin.com/profile/scootergrisen) - Dutch - Emily Hendle - [GitLab](https://gitlab.com/pundachan), [Crowdin](https://crowdin.com/profile/pandachan) +- English (UK) + - Andrew Smith - [GitLab](https://gitlab.com/espadav8), [Crowdin](https://crowdin.com/profile/espadav8) - Esperanto - Lyubomir Vasilev - [Crowdin](https://crowdin.com/profile/lyubomirv) - Estonian @@ -54,6 +56,7 @@ are very appreciative of the work done by translators and proofreaders! - Andrei Jiroh Halili - [GitLab](https://gitlab.com/ajhalili2006), [Crowdin](https://crowdin.com/profile/AndreiJirohHaliliDev2006) - French - Davy Defaud - [GitLab](https://gitlab.com/DevDef), [Crowdin](https://crowdin.com/profile/DevDef) + - Germain Gorisse - [GitLab](https://gitlab.com/ggorisse), [Crowdin](https://crowdin.com/profile/germaingorisse) - Galician - Antón Méixome - [Crowdin](https://crowdin.com/profile/meixome) - Pedro Garcia - [GitLab](https://gitlab.com/pedgarrod), [Crowdin](https://crowdin.com/profile/breaking_pitt) @@ -61,6 +64,7 @@ are very appreciative of the work done by translators and proofreaders! - Michael Hahnle - [GitLab](https://gitlab.com/mhah), [Crowdin](https://crowdin.com/profile/mhah) - Katrin Leinweber - [GitLab](https://gitlab.com/katrinleinweber), [Crowdin](https://crowdin.com/profile/katrinleinweber) - Justman10000 - [GitLab](https://gitlab.com/Justman10000), [Crowdin](https://crowdin.com/profile/Justman10000) + - Vladislav Wanner - [GitLab](https://gitlab.com/RumBugen), [Crowdin](https://crowdin.com/profile/RumBugen) - Greek - Proofreaders needed. - Hebrew diff --git a/doc/development/import_project.md b/doc/development/import_project.md index e910983997c..c63ba229921 100644 --- a/doc/development/import_project.md +++ b/doc/development/import_project.md @@ -80,6 +80,14 @@ If you're running Omnibus, run the following Rake task: gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" ``` +#### Enable verbose output + +To make the import Rake task more verbose, use the `IMPORT_DEBUG` environment variable: + +```shell +IMPORT_DEBUG=true gitlab-rake "gitlab:import_export:import[root, group/subgroup, testingprojectimport, /path/to/file.tar.gz]" +``` + #### Troubleshooting Check the common errors listed below, what they mean, and how to fix them. @@ -133,6 +141,51 @@ During import, the tarball is cached in your configured `shared_path` directory. disk has enough free space to accommodate both the cached tarball and the unpacked project files on disk. +##### Import is successful, but with a `Total number of not imported relations: XX` message, and issues are not created during the import + +If you receive a `Total number of not imported relations: XX` message, and issues +aren't created during the import, check [exceptions_json.log](../administration/logs.md#exceptions_jsonlog). +You might see an error like `N is out of range for ActiveModel::Type::Integer with limit 4 bytes`, +where `N` is the integer exceeding the 4-byte integer limit. If that's the case, you +are likely hitting the issue with rebalancing of `relative_position` field of the issues. + +The feature flag to enable the rebalance automatically was enabled on GitLab.com. +We intend to enable it by default on self-managed instances when the issue +[Rebalance issues FF rollout](https://gitlab.com/gitlab-org/gitlab/-/issues/343368) +is implemented. + +If the feature is not enabled by default on your GitLab version, run the following +commands in the [Rails console](../administration/operations/rails_console.md) as +a workaround. Replace the ID with the ID of your project you were trying to import: + +```ruby +# Check if the feature is enabled on your instance. If it is, rebalance should work automatically on your instance +Feature.enabled?(:rebalance_issues,Project.find(ID).root_namespace) + +# Check the current maximum value of relative_position +Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position) + +# Enable `rebalance_issues` feauture and check that it was successfully enabled +Feature.enable(:rebalance_issues,Project.find(ID).root_namespace) +Feature.enabled?(:rebalance_issues,Project.find(ID).root_namespace) + +# Run the rebalancing process and check if the maximum value of relative_position has changed +Issues::RelativePositionRebalancingService.new(Project.find(ID).root_namespace.all_projects).execute +Issue.where(project_id: Project.find(ID).root_namespace.all_projects).maximum(:relative_position) +``` + +Repeat the import attempt after that and check if the issues are imported successfully. + +##### Gitaly calls error when importing + +If you're attempting to import a large project into a development environment, you may see Gitaly throw an error about too many calls or invocations, for example: + +```plaintext +Error importing repository into qa-perf-testing/gitlabhq - GitalyClient#call called 31 times from single request. Potential n+1? +``` + +This is due to a [n+1 calls limit being set for development setups](gitaly.md#toomanyinvocationserror-errors). You can work around this by setting `GITALY_DISABLE_REQUEST_LIMITS=1` as an environment variable, restarting your development environment and importing again. + ### Importing via the Rails console The last option is to import a project using a Rails console: @@ -206,20 +259,6 @@ You can execute the script from the `gdk/gitlab` directory like this: bundle exec rails r /path_to_script/script.rb project_name /path_to_extracted_project request_store_enabled ``` -## Troubleshooting - -This section details known issues we've seen when trying to import a project and how to manage them. - -### Gitaly calls error when importing - -If you're attempting to import a large project into a development environment, you may see Gitaly throw an error about too many calls or invocations, for example: - -```plaintext -Error importing repository into qa-perf-testing/gitlabhq - GitalyClient#call called 31 times from single request. Potential n+1? -``` - -This is due to a [n+1 calls limit being set for development setups](gitaly.md#toomanyinvocationserror-errors). You can work around this by setting `GITALY_DISABLE_REQUEST_LIMITS=1` as an environment variable, restarting your development environment and importing again. - ## Access token setup Many of the tests also require a GitLab Personal Access Token. This is due to numerous endpoints themselves requiring authentication. diff --git a/doc/development/index.md b/doc/development/index.md index 3d5ec24d3e2..1b897db5097 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -46,307 +46,3 @@ GitLab instance, see the [Administrator documentation](../administration/index.m - [Implement design & UI elements](contributing/design.md) - [GitLab Architecture Overview](architecture.md) - [Rake tasks](rake_tasks.md) for development - -## Processes - -**Must-reads:** - -- [Guide on adapting existing and introducing new components](architecture.md#adapting-existing-and-introducing-new-components) -- [Code review guidelines](code_review.md) for reviewing code and having code - reviewed -- [Database review guidelines](database_review.md) for reviewing - database-related changes and complex SQL queries, and having them reviewed -- [Secure coding guidelines](secure_coding_guidelines.md) -- [Pipelines for the GitLab project](pipelines.md) - -Complementary reads: - -- [GitLab core team & GitLab Inc. contribution process](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md) -- [Security process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/security/developer.md#security-releases-critical-non-critical-as-a-developer) -- [Patch release process for developers](https://gitlab.com/gitlab-org/release/docs/blob/master/general/patch/process.md#process-for-developers) -- [Guidelines for implementing Enterprise Edition features](ee_features.md) -- [Adding a new service component to GitLab](adding_service_component.md) -- [Guidelines for changelogs](changelog.md) -- [Dependencies](dependencies.md) -- [Danger bot](dangerbot.md) -- [Requesting access to ChatOps on GitLab.com](chatops_on_gitlabcom.md#requesting-access) (for GitLab team members) - -### Development guidelines review - -When you submit a change to the GitLab development guidelines, who -you ask for reviews depends on the level of change. - -#### Wording, style, or link changes - -Not all changes require extensive review. For example, MRs that don't change the -content's meaning or function can be reviewed, approved, and merged by any -maintainer or Technical Writer. These can include: - -- Typo fixes. -- Clarifying links, such as to external programming language documentation. -- Changes to comply with the [Documentation Style Guide](documentation/index.md) - that don't change the intent of the documentation page. - -#### Specific changes - -If the MR proposes changes that are limited to a particular stage, group, or team, -request a review and approval from an experienced GitLab Team Member in that -group. For example, if you're documenting a new internal API used exclusively by -a given group, request an engineering review from one of the group's members. - -After the engineering review is complete, assign the MR to the -[Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) -in the modified documentation page's metadata. -If the page is not assigned to a specific group, follow the -[Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). - -#### Broader changes - -Some changes affect more than one group. For example: - -- Changes to [code review guidelines](code_review.md). -- Changes to [commit message guidelines](contributing/merge_request_workflow.md#commit-messages-guidelines). -- Changes to guidelines in [feature flags in development of GitLab](feature_flags/). -- Changes to [feature flags documentation guidelines](documentation/feature_flags.md). - -In these cases, use the following workflow: - -1. Request a peer review from a member of your team. -1. Request a review and approval of an Engineering Manager (EM) - or Staff Engineer who's responsible for the area in question: - - - [Frontend](https://about.gitlab.com/handbook/engineering/frontend/) - - [Backend](https://about.gitlab.com/handbook/engineering/) - - [Database](https://about.gitlab.com/handbook/engineering/development/database/) - - [User Experience (UX)](https://about.gitlab.com/handbook/engineering/ux/) - - [Security](https://about.gitlab.com/handbook/engineering/security/) - - [Quality](https://about.gitlab.com/handbook/engineering/quality/) - - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/) - - [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/) - - [Technical Writing](https://about.gitlab.com/handbook/engineering/ux/technical-writing/) - - You can skip this step for MRs authored by EMs or Staff Engineers responsible - for their area. - - If there are several affected groups, you may need approvals at the - EM/Staff Engineer level from each affected area. - -1. After completing the reviews, consult with the EM/Staff Engineer - author / approver of the MR. - - If this is a significant change across multiple areas, request final review - and approval from the VP of Development, the DRI for Development Guidelines, - @clefelhocz1. - -1. After all approvals are complete, assign the MR to the - [Technical Writer associated with the stage and group](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments) - in the modified documentation page's metadata. - If the page is not assigned to a specific group, follow the - [Technical Writing review process for development guidelines](https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines). - The Technical Writer may ask for additional approvals as previously suggested before merging the MR. - -### Reviewer values - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57293) in GitLab 14.1. - -As a reviewer or as a reviewee, make sure to familiarize yourself with -the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/reviewer-values/) we strive for at GitLab. - -## UX and Frontend guides - -- [GitLab Design System](https://design.gitlab.com/), for building GitLab with - existing CSS styles and elements -- [Frontend guidelines](fe_guide/index.md) -- [Emoji guide](fe_guide/emojis.md) - -## Backend guides - -### General - -- [Directory structure](directory_structure.md) -- [GitLab EventStore](event_store.md) to publish/subscribe to domain events -- [GitLab utilities](utilities.md) -- [Newlines style guide](newlines_styleguide.md) -- [Logging](logging.md) -- [Dealing with email/mailers](emails.md) -- [Kubernetes integration guidelines](kubernetes.md) -- [Permissions](permissions.md) -- [Code comments](code_comments.md) -- [Windows Development on GCP](windows.md) -- [FIPS compliance](fips_compliance.md) -- [`Gemfile` guidelines](gemfile.md) -- [Ruby upgrade guidelines](ruby_upgrade.md) - -### Things to be aware of - -- [Gotchas](gotchas.md) to avoid -- [Avoid modules with instance variables](module_with_instance_variables.md), if - possible -- [Guidelines for reusing abstractions](reusing_abstractions.md) -- [Ruby 3 gotchas](ruby3_gotchas.md) - -### Rails Framework related - -- [Routing](routing.md) -- [Rails initializers](rails_initializers.md) -- [Mass Inserting Models](mass_insert.md) -- [Issuable-like Rails models](issuable-like-models.md) -- [Issue types vs first-class types](issue_types.md) -- [DeclarativePolicy framework](policies.md) -- [Rails update guidelines](rails_update.md) - -### Debugging - -- [Pry debugging](pry_debugging.md) -- [Sidekiq debugging](../administration/troubleshooting/sidekiq.md) - -### Git specifics - -- [How Git object deduplication works in GitLab](git_object_deduplication.md) -- [Git LFS](lfs.md) - -### API - -- [API style guide](api_styleguide.md) for contributing to the API -- [GraphQL API style guide](api_graphql_styleguide.md) for contributing to the - [GraphQL API](../api/graphql/index.md) - -### GitLab components and features - -- [Developing against interacting components or features](interacting_components.md) -- [Manage feature flags](feature_flags/index.md) -- [Licensed feature availability](licensed_feature_availability.md) -- [Accessing session data](session.md) -- [How to dump production data to staging](db_dump.md) -- [Geo development](geo.md) -- [Redis guidelines](redis.md) - - [Adding a new Redis instance](redis/new_redis_instance.md) -- [Sidekiq guidelines](sidekiq/index.md) for working with Sidekiq workers -- [Working with Gitaly](gitaly.md) -- [Elasticsearch integration docs](elasticsearch.md) -- [Working with merge request diffs](diffs.md) -- [Approval Rules](approval_rules.md) -- [Repository mirroring](repository_mirroring.md) -- [Uploads development guide](uploads/index.md) -- [Auto DevOps development guide](auto_devops.md) -- [Renaming features](renaming_features.md) -- [Code Intelligence](code_intelligence/index.md) -- [Feature categorization](feature_categorization/index.md) -- [Wikis development guide](wikis.md) -- [Image scaling guide](image_scaling.md) -- [Cascading Settings](cascading_settings.md) -- [Shell commands](shell_commands.md) in the GitLab codebase -- [Value Stream Analytics development guide](value_stream_analytics.md) -- [Application limits](application_limits.md) - -### Import/Export - -- [Working with the GitHub importer](github_importer.md) -- [Import/Export development documentation](import_export.md) -- [Test Import Project](import_project.md) -- [Group migration](bulk_import.md) -- [Export to CSV](export_csv.md) - -## Language-specific guides - -### Go guides - -- [Go Guidelines](go_guide/index.md) - -### Shell Scripting guides - -- [Shell scripting standards and style guidelines](shell_scripting_guide/index.md) - -## Performance guides - -- [Performance guidelines](performance.md) for writing code, benchmarks, and - certain patterns to avoid. -- [Caching guidelines](caching.md) for using caching in Rails under a GitLab environment. -- [Merge request performance guidelines](merge_request_performance_guidelines.md) - for ensuring merge requests do not negatively impact GitLab performance -- [Profiling](profiling.md) a URL or tracking down N+1 queries using Bullet. -- [Cached queries guidelines](cached_queries.md), for tracking down N+1 queries - masked by query caching, memory profiling and why should we avoid cached - queries. - -## Database guides - -See [database guidelines](database/index.md). - -## Integration guides - -- [Integrations development guide](integrations/index.md) -- [Jira Connect app](integrations/jira_connect.md) -- [Security Scanners](integrations/secure.md) -- [Secure Partner Integration](integrations/secure_partner_integration.md) -- [How to run Jenkins in development environment](integrations/jenkins.md) -- [How to run local `Codesandbox` integration for Web IDE Live Preview](integrations/codesandbox.md) - -## Testing guides - -- [Testing standards and style guidelines](testing_guide/index.md) -- [Frontend testing standards and style guidelines](testing_guide/frontend_testing.md) - -## Refactoring guides - -- [Refactoring guidelines](refactoring_guide/index.md) - -## Deprecation guides - -- [Deprecation guidelines](deprecation_guidelines/index.md) - -## Documentation guides - -- [Writing documentation](documentation/index.md) -- [Documentation style guide](documentation/styleguide/index.md) -- [Markdown](../user/markdown.md) - -## Internationalization (i18n) guides - -- [Introduction](i18n/index.md) -- [Externalization](i18n/externalization.md) -- [Translation](i18n/translation.md) - -## Product Intelligence guides - -- [Product Intelligence guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) -- [Service Ping guide](service_ping/index.md) -- [Snowplow guide](snowplow/index.md) - -## Experiment guide - -- [Introduction](experiment_guide/index.md) - -## Build guides - -- [Building a package for testing purposes](build_test_package.md) - -## Compliance - -- [Licensing](licensing.md) for ensuring license compliance - -## Domain-specific guides - -- [CI/CD development documentation](cicd/index.md) -- [AppSec development documentation](appsec/index.md) - -## Technical Reference by Group - -- [Create: Source Code BE](backend/create_source_code_be/index.md) - -## Other Development guides - -- [Defining relations between files using projections](projections.md) -- [Reference processing](reference_processing.md) -- [Compatibility with multiple versions of the application running at the same time](multi_version_compatibility.md) -- [Features inside `.gitlab/`](features_inside_dot_gitlab.md) -- [Dashboards for stage groups](stage_group_dashboards.md) -- [Preventing transient bugs](transient/prevention-patterns.md) -- [GitLab Application SLIs](application_slis/index.md) -- [Spam protection and CAPTCHA development guide](spam_protection_and_captcha/index.md) - -## Other GitLab Development Kit (GDK) guides - -- [Run full Auto DevOps cycle in a GDK instance](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/auto_devops.md) -- [Using GitLab Runner with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/blob/main/doc/howto/runner.md) -- [Using the Web IDE terminal with the GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/web_ide_terminal_gdk_setup.md) diff --git a/doc/development/integrations/index.md b/doc/development/integrations/index.md index 604e481a809..5d1bd5ad61c 100644 --- a/doc/development/integrations/index.md +++ b/doc/development/integrations/index.md @@ -148,6 +148,7 @@ This method should return an array of hashes for each field, where the keys can | `title:` | string | false | Capitalized value of `name:` | The label for the form field. | `placeholder:` | string | false | | A placeholder for the form field. | `help:` | string | false | | A help text that displays below the form field. +| `api_only:` | boolean | false | `false` | Specify if the field should only be available through the API, and excluded from the frontend form. #### Additional keys for `type: 'checkbox'` diff --git a/doc/development/integrations/secure.md b/doc/development/integrations/secure.md index 1a51ee88c58..0a0c5e4d2a6 100644 --- a/doc/development/integrations/secure.md +++ b/doc/development/integrations/secure.md @@ -316,11 +316,12 @@ and [Container Scanning](../../user/application_security/container_scanning/inde You can find the schemas for these scanners here: -- [SAST](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json) -- [DAST](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dast-report-format.json) -- [Dependency Scanning](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dependency-scanning-report-format.json) +- [Cluster Image Scanning](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/cluster-image-scanning-report-format.json) - [Container Scanning](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/container-scanning-report-format.json) - [Coverage Fuzzing](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/coverage-fuzzing-report-format.json) +- [DAST](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dast-report-format.json) +- [Dependency Scanning](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/dependency-scanning-report-format.json) +- [SAST](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json) - [Secret Detection](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/secret-detection-report-format.json) ### Retention period for vulnerabilities @@ -360,6 +361,41 @@ Ongoing improvements to report validation are tracked [in this epic](https://git In the meantime, you can see which versions are supported in the [source code](https://gitlab.com/gitlab-org/gitlab/-/blob/08dd756429731a0cca1e27ca9d59eea226398a7d/lib/gitlab/ci/parsers/security/validators/schema_validator.rb#L9-27). +#### Validate locally + +Before running your analyzer in GitLab, you should validate the report produced by your analyzer to +ensure it complies with the declared schema version. + +Use the script below to validate JSON files against a given schema. + +```ruby +require 'bundler/inline' + +gemfile do + source 'https://rubygems.org' + gem 'json_schemer' +end + +require 'json' +require 'pathname' + +raise 'Usage: ruby script.rb <security schema file name> <report file name>' unless ARGV.size == 2 + +schema = JSONSchemer.schema(Pathname.new(ARGV[0])) +report = JSON.parse(File.open(ARGV[1]).read) +schema_validation_errors = schema.validate(report).map { |error| JSONSchemer::Errors.pretty(error) } +puts(schema_validation_errors) +``` + +1. Download the appropriate schema that matches your report type and declared version. For + example, you can find version `14.0.6` of the `container_scanning` report schema at + `https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/raw/v14.0.6/dist/container-scanning-report-format.json?inline=false`. +1. Save the Ruby script above in a file, for example, `validate.rb`. +1. Run the script, passing the schema and report file names as arguments in order. For example: + 1. Using your local Ruby interpreter: `ruby validate.rb container-scanning-format_14-0-6.json gl-container-scanning-report.json`. + 1. Using Docker: `docker run -it --rm -v $(pwd):/ci ruby:3-slim ruby /ci/validate.rb /ci/container-scanning-format_14-0-6.json /ci/gl-container-scanning-report.json` +1. Validation errors are shown on the screen. You must resolve these errors before GitLab can ingest your report. + ### Report Fields #### Version @@ -451,7 +487,7 @@ The `identifiers` array describes the detected vulnerability. An identifier obje `value` fields are used to tell if two identifiers are the same. The user interface uses the object's `name` and `url` fields to display the identifier. -It is recommended to reuse the identifiers the GitLab scanners already define: +We recommend that you use the identifiers the GitLab scanners already define: | Identifier | Type | Example value | |------------|------|---------------| @@ -479,7 +515,7 @@ Not all vulnerabilities have CVEs, and a CVE can be identified multiple times. A isn't a stable identifier and you shouldn't assume it as such when tracking vulnerabilities. The maximum number of identifiers for a vulnerability is set as 20. If a vulnerability has more than 20 identifiers, -the system saves only the first 20 of them. Note that vulnerabilities in the [Pipeline Security](../../user/application_security/security_dashboard/#view-vulnerabilities-in-a-pipeline) +the system saves only the first 20 of them. Note that vulnerabilities in the [Pipeline Security](../../user/application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline) tab do not enforce this limit and all identifiers present in the report artifact are displayed. #### Details diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md index 288c0056821..13e095b4a83 100644 --- a/doc/development/internal_api/index.md +++ b/doc/development/internal_api/index.md @@ -334,14 +334,15 @@ Example response: ## Authenticate Error Tracking requests This endpoint is called by the error tracking Go REST API application to authenticate a project. +> [Introduced](https://gitlab.com/gitlab-org/opstrace/opstrace/-/issues/1693) in GitLab 15.1. | Attribute | Type | Required | Description | |:-------------|:--------|:---------|:-------------------------------------------------------------------| | `project_id` | integer | yes | The ID of the project which has the associated key. | -| `public_key` | string | yes | The public key generated by the integrated error tracking feature. | +| `public_key` | string | yes | The [public key](../../api/error_tracking.md#error-tracking-client-keys) generated by the integrated Error Tracking feature. | ```plaintext -POST /internal/error_tracking_allowed +POST /internal/error_tracking/allowed ``` Example request: @@ -349,7 +350,7 @@ Example request: ```shell curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ --data "project_id=111&public_key=generated-error-tracking-key" \ - "http://localhost:3001/api/v4/internal/error_tracking_allowed" + "http://localhost:3001/api/v4/internal/error_tracking/allowed" ``` Example response: diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md index b4459b53efa..1159e3755e5 100644 --- a/doc/development/iterating_tables_in_batches.md +++ b/doc/development/iterating_tables_in_batches.md @@ -42,20 +42,20 @@ The API of this method is similar to `in_batches`, though it doesn't support all of the arguments that `in_batches` supports. You should always use `each_batch` _unless_ you have a specific need for `in_batches`. -## Avoid iterating over non-unique columns +## Iterating over non-unique columns -One should proceed with extra caution, and possibly avoid iterating over a column that can contain -duplicate values. When you iterate over an attribute that is not unique, even with the applied max -batch size, there is no guarantee that the resulting batches do not surpass it. The following -snippet demonstrates this situation when one attempt to select `Ci::Build` entries for users with -`id` between `1` and `10,000`, the database returns `1 215 178` matching rows. +One should proceed with extra caution. When you iterate over an attribute that is not unique, +even with the applied max batch size, there is no guarantee that the resulting batches do not +surpass it. The following snippet demonstrates this situation when one attempt to select +`Ci::Build` entries for users with `id` between `1` and `10,000`, the database returns +`1 215 178` matching rows. ```ruby [ gstg ] production> Ci::Build.where(user_id: (1..10_000)).size => 1215178 ``` -This happens because built relation is translated into the following query +This happens because the built relation is translated into the following query: ```ruby [ gstg ] production> puts Ci::Build.where(user_id: (1..10_000)).to_sql @@ -69,6 +69,27 @@ threshold does not translate to the size of the returned dataset. That happens b `n` possible values of attributes, one can't tell for sure that the number of records that contains them is less than `n`. +### Loose-index scan with `distinct_each_batch` + +When iterating over a non-unique column is necessary, use the `distinct_each_batch` helper +method. The helper uses the [loose-index scan technique](https://wiki.postgresql.org/wiki/Loose_indexscan) +(skip-index scan) to skip duplicated values within a database index. + +Example: iterating over distinct `author_id` in the Issue model + +```ruby +Issue.distinct_each_batch(column: :author_id, of: 1000) do |relation| + users = User.where(id: relation.select(:author_id)).to_a +end +``` + +The technique provides stable performance between the batches regardless of the data distribution. +The `relation` object returns an ActiveRecord scope where only the given `column` is available. +Other columns are not loaded. + +The underlying database queries use recursive CTEs, which adds extra overhead. We therefore advise to use +smaller batch sizes than those used for a standard `each_batch` iteration. + ## Column definition `EachBatch` uses the primary key of the model by default for the iteration. This works most of the diff --git a/doc/development/licensed_feature_availability.md b/doc/development/licensed_feature_availability.md index 09c32fc4244..21b07ae89b5 100644 --- a/doc/development/licensed_feature_availability.md +++ b/doc/development/licensed_feature_availability.md @@ -61,3 +61,12 @@ before_action do push_licensed_feature(:feature_symbol, project) end ``` + +## Allow use of licensed EE features + +To enable plans per namespace turn on the `Allow use of licensed EE features` option from the settings page. +This will make licensed EE features available to projects only if the project namespace's plan includes the feature +or if the project is public. To enable it: + +1. If you are developing locally, follow the steps in [simulate SaaS](ee_features.md#act-as-saas) to make the option available. +1. Select Admin > Settings > General > "Account and limit" and enable "Allow use of licensed EE features". diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index c9b59ba66b5..e0e21319f47 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -90,6 +90,14 @@ Keep in mind that all durations should be measured against GitLab.com. | Post-deployment migrations | `<= 10 minutes` | A valid exception are schema changes, since they must not happen in background migrations. | | Background migrations | `> 10 minutes` | Since these are suitable for larger tables, it's not possible to set a precise timing guideline, however, any single query must stay below [`1 second` execution time](query_performance.md#timing-guidelines-for-queries) with cold caches. | +## Decide which database to target + +GitLab connects to two different Postgres databases: `main` and `ci`. This split can affect migrations +as they may run on either or both of these databases. + +Read [Migrations for Multiple databases](database/migrations_for_multiple_databases.md) to understand if or how +a migration you add should account for this. + ## Create a regular schema migration To create a migration you can use the following Rails generator: @@ -569,6 +577,12 @@ class MyMigration < Gitlab::Database::Migration[2.0] end ``` +Verify the index is not being used anymore with this Thanos query: + +```sql +sum(rate(pg_stat_user_indexes_idx_tup_read{env="gprd", indexrelname="index_ci_name", type="patroni-ci"}[5m])) +``` + Note that it is not necessary to check if the index exists prior to removing it, however it is required to specify the name of the index that is being removed. This can be done either by passing the name diff --git a/doc/development/omnibus.md b/doc/development/omnibus.md index b62574e34e5..4e2f2b0c763 100644 --- a/doc/development/omnibus.md +++ b/doc/development/omnibus.md @@ -21,7 +21,7 @@ For example, the `git` user is allowed to write in the `log/` directory, in `public/uploads`, and they are allowed to rewrite the `db/structure.sql` file. In other cases, the reconfigure script tricks GitLab into not trying to write a -file. For instance, GitLab will generate a `.secret` file if it cannot find one +file. For instance, GitLab generates a `.secret` file if it cannot find one and write it to the Rails root. In the Omnibus packages, reconfigure writes the `.secret` file first, so that GitLab never tries to write it. diff --git a/doc/development/packages/debian_repository.md b/doc/development/packages/debian_repository.md new file mode 100644 index 00000000000..a417ced2e65 --- /dev/null +++ b/doc/development/packages/debian_repository.md @@ -0,0 +1,151 @@ +--- +stage: Package +group: Package +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Debian Repository + +This guide explains: + +1. A basic overview of how Debian packages are structured +1. What package managers, clients, and tools are used to manage Debian packages +1. How the GitLab Debian repository functions + +## Debian package basics + +There are two types of [Debian packages](https://www.debian.org/doc/manuals/debian-faq/pkg-basics.en.html): binary and source. + +- **Binary** - These are usually `.deb` files and contain executables, config files, and other data. A binary package must match your OS or architecture since it is already compiled. These are usually installed using `dpkg`. Dependencies must already exist on the system when installing a binary package. +- **Source** - These are usual made up of `.dsc` files and `.gz` files. A source package is compiled on your system. These are fetched and installed with [`apt`](https://manpages.debian.org/bullseye/apt/apt.8.en.html), which then uses `dpkg` after the package is compiled. When you use `apt`, it will fetch and install the necessary dependencies. + +The `.deb` file follows the naming convention `<PackageName>_<VersionNumber>-<DebianRevisionNumber>_<DebianArchitecture>.deb` + +It includes a `control file` that contains metadata about the package. You can view the control file by using `dpkg --info <deb_file>` + +The [`.changes` file](https://www.debian.org/doc/debian-policy/ch-controlfields.html#debian-changes-files-changes) is used to tell the Debian repository how to process updates to packages. It contains a variety of metadata for the package, including architecture, distribution, and version. In addition to the metadata, they contain three lists of checksums: `sha1`, `sha256`, and `md5` in the `Files` section. Refer to [sample_1.2.3~alpha2_amd64.changes](https://gitlab.com/gitlab-org/gitlab/-/blob/dd1e70d3676891025534dc4a1e89ca9383178fe7/spec/fixtures/packages/debian/sample_1.2.3~alpha2_amd64.changes) for an example of how these files are structured. + +## How do people get Debian packages? + +While you can download a single `.deb` file and install it with [`dpkg`](https://manpages.debian.org/bullseye/dpkg/dpkg.1.en.html), most users consume Debian packages with [`apt`](https://manpages.debian.org/bullseye/apt/apt.8.en.html) using `apt-get`. `apt` wraps `dpkg`, adding dependency management and compilation. + +## How do people publish Debian packages? + +It is not uncommon to use `curl` to publish packages depending on the type of Debian repository you are working with. However, `dput-ng` is the best tool to use as it will upload the relevant files based on the `.changes` file. + +## What is all this distribution business? + +When it comes to Debian, packages don't exist on their own. They belong to a _distribution_. This can mean many things, but the main thing to note is users are used to having to specify the distribution. + +## What does a Debian Repository look like? + +- A [Debian repository](https://wiki.debian.org/DebianRepository) is made up of many releases. +- Each release is given a **codename**. For the public Debian repository, these are things like "bullseye" and "jesse". + - There is also the concept of **suites** which are essentially aliases of codenames synonymous with release channels like "stable" and "edge". +- Each release has many **components**. In the public repository, these are "main", "contrib", and "non-free". +- Each release has many **architectures** such as "amd64", "arm64", or "i386". +- Each release has a signed **Release** file (see below about [GPG signing](#what-are-gpg-keys-and-what-are-signed-releases)) + +A standard directory-based Debian repository would be organized as: + +```plaintext +dists\ + |--jessie/ + |--bullseye\ + |Changelog + |Release + |InRelease + |Release.gpg + |--main\ + |--amd64\ + |--arm64\ + |--contrib\ + |--non-free\ +pool\ + |--this is where the .deb files for all releases live +``` + +You can explore a mirror of the public Debian repository here: <http://ftp.us.debian.org/debian/> + +In the public Debian repository, the entire directory structure, release files, GPG keys, and other files are all generated by a series of scripts called the [Debian Archive Kit, or dak](https://salsa.debian.org/ftp-team/dak). + +In the GitLab Debian repository, we don't deal with specific file directories. Instead, we use code and an underlying [PostgreSQL database to organize the relationships](structure.md#debian-packages) between these different pieces. + +## What does a Debian Repository do? + +The Debian community created many package repository systems before things like object storage existed, and they used FTP to upload artifacts to a remote server. Most current package repositories and registries are just directories on a server somewhere. Packages added to the [official Debian distribution](https://www.debian.org/distrib/packages) exist in a central public repository that a group of open source maintainers curates. The package maintainers use the [Debian Archive Kit, or dak](https://salsa.debian.org/ftp-team/dak) scripts to generate release files and do other maintenance tasks. So, in addition to storing and serving files, a complete Debian repository needs to accomplish the same behavior that dak provides. This behavior is what the GitLab Debian registry aims to do. + +## What are GPG keys, and what are signed releases + +A [GPG key](https://www.gnupg.org/) is a public/private key pair for secure data transmission. Similar to an SSH key, there is a private and public key. Whoever has the _public key can encrypt data_, and whoever has the _private key can decrypt data_ that was encrypted using the public key. You can also use GPG keys to sign data. Whoever has the private key can sign data or a file, and whoever has the public key can then check the signature and trust it came from the person with the matching private key. + +We use GPG to sign the release file for the Debian packages. The release file is an index of all packages within a given distribution and their respective digests. + +In the GitLab Debian registry, a background process generates a new release file whenever a user publishes a new package to their Debian repository. A GPG key is created for each distribution. If a user requests a release for that distribution, they can request the signed version and the public GPG key to verify the authenticity of that release file. + +## GitLab repository internals + +When a [file upload](../../api/packages/debian.md#upload-a-package-file) occurs: + +1. A new "incoming" package record is found or created. All new files are assigned to the "incoming" package. It is a holding area used until we know what package the file is actually associated with. +1. A new "unknown" file is stored. It is unknown because we do not yet know if this file belongs to an existing package or not. + +Once we know which package the file belongs to, it is associated with that package, and the "incoming" package is removed if no more files remain. The "unknown" status of the file is updated to the correct file type. + +Next, if the file is a `.changes` format: + +1. The `.changes` file is parsed and any files listed within it are updated. All uploaded non-`.changes` files are correctly associated with various distributions and packages. +1. The `::Packages::Debian::GenerateDistributionWorker` and thus `::Packages::Debian::GenerateDistributionService` are run. + 1. Component files are created or updated. Since we just updated package files that were listed in the `.changes` file, we now check the component/architecture files based on the changed checksum values. + 1. A new release is generated: + 1. A new GPG key is generated if one does not already exist for the distribution + 1. A [Release file](https://wiki.debian.org/DebianRepository/Format#A.22Release.22_files) is written, signed by the GPG key, and then stored. + 1. Old component files are destroyed. + +This diagram shows the path taken after a file is uploaded to the Debian API: + +```mermaid +sequenceDiagram + Client->>+DebianProjectPackages: PUT projects/:id/packages/debian/:file_name + DebianProjectPackages->>+FindOrCreateIncomingService: Create "incoming" package + DebianProjectPackages->>+CreatePackageFileService: Create "unknown" file + Note over DebianProjectPackages: If `.changes` file + DebianProjectPackages->>+ProcessChangesWorker: Schedule worker to process the file + DebianProjectPackages->>+Client: 202 Created + ProcessChangesWorker->>+ProcessChangesService: Start service + ProcessChangesService->>+ExtractChangesMetadataService: Extract changesmetadata + ExtractChangesMetadataService->>+ExtractMetadataService: Extract file metadata + ExtractMetadataService->>+ParseDebian822Service: run `dpkg --field` to get control file + ExtractMetadataService->>+ExtractDebMetadataService: If .deb or .udeb + ExtractDebMetadataService->>+ParseDebian822Service: run `dpkg --field` to get control file + ParseDebian822Service-->>-ExtractDebMetadataService: Parse String as Debian RFC822 control data format + ExtractDebMetadataService-->>-ExtractMetadataService: Return the parsed control file + ExtractMetadataService->>+ParseDebian822Service: if .dsc, .changes, or buildinfo + ParseDebian822Service-->>-ExtractMetadataService: Parse String as Debian RFC822 control data format + ExtractMetadataService-->>-ExtractChangesMetadataService: Parse Metadata file + ExtractChangesMetadataService-->>-ProcessChangesService: Return list of files and hashes from the .changes file + loop process files listed in .changes + ProcessChangesService->>+ExtractMetadataService: Process file + ExtractMetadataService->>+ParseDebian822Service: run `dpkg --field` to get control file + ExtractMetadataService->>+ExtractDebMetadataService: If .deb or .udeb + ExtractDebMetadataService->>+ParseDebian822Service: run `dpkg --field` to get control file + ParseDebian822Service-->>-ExtractDebMetadataService: Parse String as Debian RFC822 control data format + ExtractDebMetadataService-->>-ExtractMetadataService: Return the parsed control file + ExtractMetadataService->>+ParseDebian822Service: if .dsc, .changes, or buildinfo + ParseDebian822Service-->>-ExtractMetadataService: Parse String as Debian RFC822 control data format + ExtractMetadataService-->>-ProcessChangesService: Use parsed metadata to update "unknown" (or known) file + end + ProcessChangesService->>+GenerateDistributionWorker: Find distribution and start service + GenerateDistributionWorker->>+GenerateDistributionService: Generate distribution + GenerateDistributionService->>+GenerateDistributionService: generate component files based on new archs and updates from .changes + GenerateDistributionService->>+GenerateDistributionKeyService: generate GPG key for distribution + GenerateDistributionKeyService-->>-GenerateDistributionService: GPG key + GenerateDistributionService-->>-GenerateDistributionService: Generate distribution file + GenerateDistributionService->>+SignDistributionService: Sign release file with GPG key + SignDistributionService-->>-GenerateDistributionService: Save the signed release file + GenerateDistributionWorker->>+GenerateDistributionService: destroy no longer used component files +``` + +### Distributions + +You must create a distribution before publishing a package to it. When you create or update a distribution using the project or group distribution API, in addition to creating the initial backing records in the database, the `GenerateDistributionService` run as shown in the above sequence diagram. diff --git a/doc/development/packages/structure.md b/doc/development/packages/structure.md index a2716232b11..f8d9da2cc73 100644 --- a/doc/development/packages/structure.md +++ b/doc/development/packages/structure.md @@ -39,7 +39,6 @@ erDiagram projects }|--|| namespaces : "" packages_packages }|--|| projects : "" packages_package_files }o--|| packages_packages : "" - package_debian_file_metadatum |o--|| packages_package_files : "" packages_debian_group_architectures }|--|| packages_debian_group_distributions : "" packages_debian_group_component_files }|--|| packages_debian_group_components : "" packages_debian_group_component_files }|--|| packages_debian_group_architectures : "" diff --git a/doc/development/performance.md b/doc/development/performance.md index 6d0b833a2da..d7cbef0a211 100644 --- a/doc/development/performance.md +++ b/doc/development/performance.md @@ -26,10 +26,10 @@ consistent performance of GitLab. Refer to the [Index](#performance-documentatio - Frontend: - [Performance guidelines](../development/fe_guide/performance.md) - [Performance dashboards and monitoring guidelines](../development/new_fe_guide/development/performance.md) - - [Browser performance testing guidelines](../user/project/merge_requests/browser_performance_testing.md) + - [Browser performance testing guidelines](../ci/testing/browser_performance_testing.md) - [`gdk measure` and `gdk measure-workflow`](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/gdk_commands.md#measure-performance) - QA: - - [Load performance testing](../user/project/merge_requests/load_performance_testing.md) + - [Load performance testing](../ci/testing/load_performance_testing.md) - [GitLab Performance Tool project](https://gitlab.com/gitlab-org/quality/performance) - [Review apps performance metrics](../development/testing_guide/review_apps.md#performance-metrics) - Monitoring & Overview: @@ -581,7 +581,7 @@ called `memory-on-boot`. ([Read an example job.](https://gitlab.com/gitlab-org/g You may find the results: - On the merge request **Overview** tab, in the merge request reports area, in the - **Metrics Reports** [dropdown list](../ci/metrics_reports.md). + **Metrics Reports** [dropdown list](../ci/testing/metrics_reports.md). - In the `memory-on-boot` artifacts for a full report and a dependency breakdown. `derailed_benchmarks` also provides other methods to investigate memory. To learn more, diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 436977a7f38..2bf1e5a315a 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -37,7 +37,7 @@ flowchart LR subgraph backend be["Backend code"]--tested with-->rspec end - + be--generates-->fixtures["frontend fixtures"] fixtures--used in-->jest ``` @@ -171,7 +171,7 @@ Our current RSpec tests parallelization setup is as follows: 1. The `retrieve-tests-metadata` job in the `prepare` stage ensures we have a `knapsack/report-master.json` file: - The `knapsack/report-master.json` file is fetched from the latest `main` pipeline which runs `update-tests-metadata` - (for now it's the 2-hourly scheduled master pipeline), if it's not here we initialize the file with `{}`. + (for now it's the 2-hourly `maintenance` scheduled master pipeline), if it's not here we initialize the file with `{}`. 1. Each `[rspec|rspec-ee] [migration|unit|integration|system|geo] n m` job are run with `knapsack rspec` and should have an evenly distributed share of tests: - It works because the jobs have access to the `knapsack/report-master.json` @@ -275,6 +275,19 @@ rather than from the default branch `main-jh`. NOTE: For now, CI will try to fetch the branch on the [GitLab JH mirror](https://gitlab.com/gitlab-org/gitlab-jh-mirrors/gitlab), so it might take some time for the new JH branch to propagate to the mirror. +## Ruby 3.0 jobs + +You can add the `pipeline:run-in-ruby3` label to the merge request to switch +the Ruby version used for running the whole test suite to 3.0. When you do +this, the test suite will no longer run in Ruby 2.7 (default), and an +additional job `verify-ruby-2.7` will also run and always fail to remind us to +remove the label and run in Ruby 2.7 before merging the merge request. + +This should let us: + +- Test changes for Ruby 3.0 +- Make sure it will not break anything when it's merged into the default branch + ## `undercover` RSpec test > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/74859) in GitLab 14.6. @@ -292,20 +305,20 @@ fail. ### Troubleshooting `rspec:undercoverage` failures The `rspec:undercoverage` job has [known bugs](https://gitlab.com/groups/gitlab-org/-/epics/8254) -that can cause false positive failures. You can locally test coverage locally to determine if it's +that can cause false positive failures. You can test coverage locally to determine if it's safe to apply `~"pipeline:skip-undercoverage"`. For example, using `<spec>` as the name of the test causing the failure: 1. Run `SIMPLECOV=1 bundle exec rspec <spec>`. 1. Run `scripts/undercoverage`. -If these commands return `undercover: ✅ No coverage is missing in latest changes` then you can apply `~"pipeline:skip-undercoverage"` to bypass pipeline failures. +If these commands return `undercover: ✅ No coverage is missing in latest changes` then you can apply `~"pipeline:skip-undercoverage"` to bypass pipeline failures. ## Ruby versions testing Our test suite runs against Ruby 2 in merge requests and default branch pipelines. -We do run our test suite against Ruby 3 on 2-hourly scheduled pipelines, as GitLab.com will soon run on Ruby 3. +We also run our test suite against Ruby 3 on another 2-hourly scheduled pipelines, as GitLab.com will soon run on Ruby 3. ## PostgreSQL versions testing @@ -318,12 +331,13 @@ We also run our test suite against PG11 upon specific database library changes i ### Current versions testing -| Where? | PostgreSQL version | -| ------ | ------------------ | -| MRs | 12, 11 for DB library changes | -| `main` (non-scheduled pipelines) | 12, 11 for DB library changes | -| 2-hourly scheduled pipelines | 12, 11 for DB library changes | -| `nightly` scheduled pipelines | 12, 11, 13 | +| Where? | PostgreSQL version | Ruby version | +| ------ | ------------------ | ------------ | +| Merge requests | 12 (default version), 11 for DB library changes | 2.7 (default version) | +| `master` branch commits | 12 (default version), 11 for DB library changes | 2.7 (default version) | +| `maintenance` scheduled pipelines (every 2 hours at even hour) | 12 (default version), 11 for DB library changes | 2.7 (default version) | +| `maintenance` scheduled pipelines (every 2 hours at odd hour) | 12 (default version), 11 for DB library changes | 3.0 (set in the schedule variables) | +| `nightly` scheduled pipelines | 12 (default version), 11, 13 | 2.7 (default version) | ### Long-term plan @@ -618,7 +632,7 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/yaml_optimizat | `if-default-refs` | Matches if the pipeline is for `master`, `main`, `/^[\d-]+-stable(-ee)?$/` (stable branches), `/^\d+-\d+-auto-deploy-\d+$/` (auto-deploy branches), `/^security\//` (security branches), merge requests, and tags. | Note that jobs aren't created for branches with this default configuration. | | `if-master-refs` | Matches if the current branch is `master` or `main`. | | | `if-master-push` | Matches if the current branch is `master` or `main` and pipeline source is `push`. | | -| `if-master-schedule-2-hourly` | Matches if the current branch is `master` or `main` and pipeline runs on a 2-hourly schedule. | | +| `if-master-schedule-maintenance` | Matches if the current branch is `master` or `main` and pipeline runs on a 2-hourly schedule. | | | `if-master-schedule-nightly` | Matches if the current branch is `master` or `main` and pipeline runs on a nightly schedule. | | | `if-auto-deploy-branches` | Matches if the current branch is an auto-deploy one. | | | `if-master-or-tag` | Matches if the pipeline is for the `master` or `main` branch or for a tag. | | @@ -660,6 +674,7 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/yaml_optimizat | `code-backstage-patterns` | Combination of `code-patterns` and `backstage-patterns`. | | `code-qa-patterns` | Combination of `code-patterns` and `qa-patterns`. | | `code-backstage-qa-patterns` | Combination of `code-patterns`, `backstage-patterns`, and `qa-patterns`. | +| `static-analysis-patterns` | Only create jobs for Static Analytics configuration-related changes. | ## Performance @@ -704,7 +719,7 @@ This works well for the following reasons: - `.yarn-cache` - `.assets-compile-cache` (the key includes `${NODE_ENV}` so it's actually two different caches). 1. These cache definitions are composed of [multiple atomic caches](../ci/caching/index.md#use-multiple-caches). -1. Only the following jobs, running in 2-hourly scheduled pipelines, are pushing (that is, updating) to the caches: +1. Only the following jobs, running in 2-hourly `maintenance` scheduled pipelines, are pushing (that is, updating) to the caches: - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-gitaly-binaries-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-rubocop-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). diff --git a/doc/development/product_qualified_lead_guide/index.md b/doc/development/product_qualified_lead_guide/index.md index 25634876aef..90b8d905264 100644 --- a/doc/development/product_qualified_lead_guide/index.md +++ b/doc/development/product_qualified_lead_guide/index.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Conversion +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/rails_initializers.md b/doc/development/rails_initializers.md index 68f3c07e45a..fca24cf8c01 100644 --- a/doc/development/rails_initializers.md +++ b/doc/development/rails_initializers.md @@ -29,12 +29,18 @@ query) from an initializer means that tasks like `db:drop`, and `db:test:prepare` will fail because an active session prevents the database from being dropped. -To help detect when database connections are opened from initializers, we now -warn in `STDERR`. For example: +To prevent this, we stop database connections from being opened during +routes loading. Doing will result in an error: ```shell -DEPRECATION WARNING: Database connection should not be called during initializers (called from block in <module:HasVariable> at app/models/concerns/ci/has_variable.rb:22) +RuntimeError: + Database connection should not be called during initializers. +# ./config/initializers/00_connection_logger.rb:15:in `new_client' +# ./lib/gitlab/database/load_balancing/load_balancer.rb:112:in `block in read_write' +# ./lib/gitlab/database/load_balancing/load_balancer.rb:184:in `retry_with_backoff' +# ./lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write' +# ./lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer' +# ./lib/gitlab/database/load_balancing/connection_proxy.rb:89:in `method_missing' +# ./config/routes.rb:10:in `block in <main>' +# ./config/routes.rb:9:in `<main>' ``` - -If you wish to print out the full backtrace, set the -`DEBUG_INITIALIZER_CONNECTIONS` environment variable. diff --git a/doc/development/rake_tasks.md b/doc/development/rake_tasks.md index 13c4bdaedca..c13f1195df3 100644 --- a/doc/development/rake_tasks.md +++ b/doc/development/rake_tasks.md @@ -313,11 +313,11 @@ run: ```shell # Validate all queries -bundle exec rake gitlab::graphql:validate +bundle exec rake gitlab:graphql:validate # Validate one query -bundle exec rake gitlab::graphql:validate[path/to/query.graphql] +bundle exec rake gitlab:graphql:validate[path/to/query.graphql] # Validate a directory -bundle exec rake gitlab::graphql:validate[path/to/queries] +bundle exec rake gitlab:graphql:validate[path/to/queries] ``` This prints out a report with an entry for each query, explaining why @@ -335,11 +335,11 @@ Usage: ```shell # Analyze all queries -bundle exec rake gitlab::graphql:analyze +bundle exec rake gitlab:graphql:analyze # Analyze one query -bundle exec rake gitlab::graphql:analyze[path/to/query.graphql] +bundle exec rake gitlab:graphql:analyze[path/to/query.graphql] # Analyze a directory -bundle exec rake gitlab::graphql:analyze[path/to/queries] +bundle exec rake gitlab:graphql:analyze[path/to/queries] ``` This prints out a report for each query, including the complexity @@ -393,3 +393,21 @@ The following command combines the intent of [Update GraphQL documentation and s ```shell bundle exec rake gitlab:graphql:update_all ``` + +## Update OpenAPI client for Error Tracking feature + +NOTE: +This Rake task needs `docker` to be installed. + +To update generated code for OpenAPI client located in +`vendor/gems/error_tracking_open_api` run the following commands: + +```shell +# Run rake task +bundle exec rake gems:error_tracking_open_api:generate + +# Review and test the changes + +# Commit the changes +git commit -m 'Update ErrorTrackingOpenAPI from OpenAPI definition' vendor/gems/error_tracking_open_api +``` diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index ccf82dc6c77..f3eb1ebcc0c 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -190,6 +190,10 @@ Everything in `app/finders`, typically used for retrieving data from a database. Finders can not reuse other finders in an attempt to better control the SQL queries they produce. +Finders' `execute` method should return `ActiveRecord::Relation`. Exceptions +can be added to `spec/support/finder_collection_allowlist.yml`. +See [`#298771`](https://gitlab.com/gitlab-org/gitlab/-/issues/298771) for more details. + ### Presenters Everything in `app/presenters`, used for exposing complex data to a Rails view, diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index d8e2352bd93..9048da77071 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -1278,3 +1278,31 @@ This sensitive data must be handled carefully to avoid leaks which could lead to - Avoid sending credentials in URL parameters, as these can be more easily logged inadvertently during transit. In the event of credential leak through an MR, issue, or any other medium, [reach out to SIRT team](https://about.gitlab.com/handbook/engineering/security/security-operations/sirt/#-engaging-sirt). + +## Serialization + +Serialization of active record models can leak sensitive attributes if they are not protected. + +Using the [`prevent_from_serialization`](https://gitlab.com/gitlab-org/gitlab/-/blob/d7b85128c56cc3e669f72527d9f9acc36a1da95c/app/models/concerns/sensitive_serializable_hash.rb#L11) +method protects the attributes when the object is serialized with `serializable_hash`. +When an attribute is protected with `prevent_from_serialization`, it is not included with +`serializable_hash`, `to_json`, or `as_json`. + +For more guidance on serialization: + +- [Why using a serializer is important](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/serializers/README.md#why-using-a-serializer-is-important). +- Always use [Grape entities](../../ee/development/api_styleguide.md#entities) for the API. + +To `serialize` an `ActiveRecord` column: + +- You can use `app/serializers`. +- You cannot use `to_json / as_json`. +- You cannot use `serialize :some_colum`. + +### Serialization example + +The following is an example used for the [`TokenAuthenticatable`](https://gitlab.com/gitlab-org/gitlab/-/blob/9b15c6621588fce7a80e0438a39eeea2500fa8cd/app/models/concerns/token_authenticatable.rb#L30) class: + +```ruby +prevent_from_serialization(*strategy.token_fields) if respond_to?(:prevent_from_serialization) +``` diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 6948eb20e00..3263ba6458e 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -830,7 +830,7 @@ However, it has the following limitations: ## Aggregated metrics -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45979) in GitLab 13.6. WARNING: This feature is intended solely for internal GitLab use. diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index e776b78b710..cd8af3e9152 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -113,15 +113,15 @@ sequenceDiagram 1. Finally, the timing metadata information that is used for diagnostic purposes is submitted to the Versions application. It consists of a list of metric identifiers and the time it took to calculate the metrics: - > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37911) in GitLab 15.0 [with a flag(../../user/feature_flags.md), enabled by default. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `measure_service_ping_metric_collection`. -On GitLab.com, this feature is available. + > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37911) in GitLab 15.0 [with a flag(../../user/feature_flags.md), enabled by default. + > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/295289) in GitLab 15.2. [Feature flag `measure_service_ping_metric_collection`](https://gitlab.com/gitlab-org/gitlab/-/issues/358128) removed. ```ruby - {"metadata"=> - {"metrics"=> + { + "metadata"=> + { + "uuid"=>"0000000-0000-0000-0000-000000000000", + "metrics"=> [{"name"=>"version", "time_elapsed"=>1.1811964213848114e-05}, {"name"=>"installation_type", "time_elapsed"=>0.00017242692410945892}, {"name"=>"license_billable_users", "time_elapsed"=>0.009520471096038818}, @@ -133,9 +133,7 @@ On GitLab.com, this feature is available. {"name"=>"counts.clusters_platforms_user", "time_elapsed"=>0.06410990096628666}, {"name"=>"counts.clusters_management_project", - "time_elapsed"=>0.24020783510059118}, - {"name"=>"counts.clusters_integrations_elastic_stack", - "time_elapsed"=>0.03484998410567641} + "time_elapsed"=>0.24020783510059118} ] } } @@ -163,25 +161,6 @@ We also collect metrics specific to [Geo](../../administration/geo/index.md) sec ] ``` -### Enable or disable service ping metadata reporting - -Service Ping timing metadata reporting is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:measure_service_ping_metric_collection) -``` - -To disable it: - -```ruby -Feature.disable(:measure_service_ping_metric_collection) -``` - ## Implementing Service Ping See the [implement Service Ping](implement.md) guide. @@ -200,6 +179,7 @@ The following is example content of the Service Ping payload. "recorded_at": "2020-04-17T07:43:54.162+00:00", "edition": "EEU", "license_md5": "00000000000000000000000000000000", + "license_sha256: "0000000000000000000000000000000000000000000000000000000000000000", "license_id": null, "historical_max_users": 999, "licensee": { diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index fee3bb571c2..2adba5d8095 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -207,7 +207,7 @@ description: GitLab instance unique identifier product_category: collection product_section: growth product_stage: growth -product_group: group::product intelligence +product_group: product_intelligence value_type: string status: active milestone: 9.1 @@ -240,15 +240,17 @@ The generator takes a list of key paths and 3 options as arguments. It creates m ```shell bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d --class_name=CountIssues -create config/metrics/counts_7d/issues.yml +// Creates 1 file +// create config/metrics/counts_7d/issues.yml ``` **Multiple metrics example** ```shell bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d --class_name=CountUsersCreatingIssues -create config/metrics/counts_7d/issues.yml -create config/metrics/counts_7d/users.yml +// Creates 2 files +// create config/metrics/counts_7d/issues.yml +// create config/metrics/counts_7d/users.yml ``` NOTE: @@ -256,7 +258,8 @@ To create a metric definition used in EE, add the `--ee` flag. ```shell bundle exec rails generate gitlab:usage_metric_definition counts.issues --ee --dir=7d --class_name=CountUsersCreatingIssues -create ee/config/metrics/counts_7d/issues.yml +// Creates 1 file +// create ee/config/metrics/counts_7d/issues.yml ``` ### Metrics added dynamic to Service Ping payload @@ -265,20 +268,35 @@ The [Redis HLL metrics](implement.md#known-events-are-added-automatically-in-ser A YAML metric definition is required for each metric. A dedicated generator is provided to create metric definitions for Redis HLL events. -The generator takes `category` and `event` arguments, as the root key is `redis_hll_counters`, and creates two metric definitions for weekly and monthly time frames: +The generator takes `category` and `events` arguments, as the root key is `redis_hll_counters`, and creates two metric definitions for each of the events (for weekly and monthly time frames): + +**Single metric example** ```shell bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues count_users_closing_issues -create config/metrics/counts_7d/count_users_closing_issues_weekly.yml -create config/metrics/counts_28d/count_users_closing_issues_monthly.yml +// Creates 2 files +// create config/metrics/counts_7d/count_users_closing_issues_weekly.yml +// create config/metrics/counts_28d/count_users_closing_issues_monthly.yml +``` + +**Multiple metrics example** + +```shell +bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues count_users_closing_issues count_users_reopening_issues +// Creates 4 files +// create config/metrics/counts_7d/count_users_closing_issues_weekly.yml +// create config/metrics/counts_28d/count_users_closing_issues_monthly.yml +// create config/metrics/counts_7d/count_users_reopening_issues_weekly.yml +// create config/metrics/counts_28d/count_users_reopening_issues_monthly.yml ``` To create a metric definition used in EE, add the `--ee` flag. ```shell bundle exec rails generate gitlab:usage_metric_definition:redis_hll issues users_closing_issues --ee -create config/metrics/counts_7d/i_closed_weekly.yml -create config/metrics/counts_28d/i_closed_monthly.yml +// Creates 2 files +// create config/metrics/counts_7d/i_closed_weekly.yml +// create config/metrics/counts_28d/i_closed_monthly.yml ``` ## Metrics Dictionary diff --git a/doc/development/service_ping/metrics_instrumentation.md b/doc/development/service_ping/metrics_instrumentation.md index 4fd03eea84f..e1c51713f3c 100644 --- a/doc/development/service_ping/metrics_instrumentation.md +++ b/doc/development/service_ping/metrics_instrumentation.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/metrics_lifecycle.md b/doc/development/service_ping/metrics_lifecycle.md index c9cc9a4f2d2..28f77b6f587 100644 --- a/doc/development/service_ping/metrics_lifecycle.md +++ b/doc/development/service_ping/metrics_lifecycle.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/performance_indicator_metrics.md b/doc/development/service_ping/performance_indicator_metrics.md index 48c123171fa..bdd4c319d41 100644 --- a/doc/development/service_ping/performance_indicator_metrics.md +++ b/doc/development/service_ping/performance_indicator_metrics.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/review_guidelines.md b/doc/development/service_ping/review_guidelines.md index ee2d8f4f4a1..4ce5b2d577c 100644 --- a/doc/development/service_ping/review_guidelines.md +++ b/doc/development/service_ping/review_guidelines.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/service_ping/troubleshooting.md b/doc/development/service_ping/troubleshooting.md index 2764ef41f98..29ab334f867 100644 --- a/doc/development/service_ping/troubleshooting.md +++ b/doc/development/service_ping/troubleshooting.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -26,6 +26,16 @@ You can use [this query](https://gitlab.com/gitlab-org/gitlab/-/issues/347298#no For results about an investigation conducted into an unexpected drop in Service ping Payload events volume, see [this issue](https://gitlab.com/gitlab-data/analytics/-/issues/11071). +### Troubleshoot VersionApp layer + +Check if the [export jobs](https://gitlab.com/gitlab-services/version-gitlab-com#data-export-using-pipeline-schedules) are successful. + +Check [Service Ping errors](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health?widget=14609989&udv=0) in the [Service Ping Health Dahsboard](https://app.periscopedata.com/app/gitlab/968489/Product-Intelligence---Service-Ping-Health). + +### Troubleshoot Google Storage layer + +Check if the files are present in [Google Storage](https://console.cloud.google.com/storage/browser/cloudsql-gs-production-efd5e8-cloudsql-exports;tab=objects?project=gs-production-efd5e8&prefix=&forceOnObjectsSortingFiltering=false). + ### Troubleshoot the data warehouse layer Reach out to the [Data team](https://about.gitlab.com/handbook/business-technology/data-team/) to ask about current state of data warehouse. On their handbook page there is a [section with contact details](https://about.gitlab.com/handbook/business-technology/data-team/#how-to-connect-with-us). diff --git a/doc/development/service_ping/usage_data.md b/doc/development/service_ping/usage_data.md index a25ad5f62be..a659bbf2265 100644 --- a/doc/development/service_ping/usage_data.md +++ b/doc/development/service_ping/usage_data.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/sidekiq/compatibility_across_updates.md b/doc/development/sidekiq/compatibility_across_updates.md index 35f4b88351e..96a3573d11a 100644 --- a/doc/development/sidekiq/compatibility_across_updates.md +++ b/doc/development/sidekiq/compatibility_across_updates.md @@ -142,7 +142,10 @@ When renaming queues, use the `sidekiq_queue_migrate` helper migration method in a **post-deployment migration**: ```ruby -class MigrateTheRenamedSidekiqQueue < Gitlab::Database::Migration[1.0] +class MigrateTheRenamedSidekiqQueue < Gitlab::Database::Migration[2.0] + restrict_gitlab_migration gitlab_schema: :gitlab_main + disable_ddl_transaction! + def up sidekiq_queue_migrate 'old_queue_name', to: 'new_queue_name' end diff --git a/doc/development/snowplow/event_dictionary_guide.md b/doc/development/snowplow/event_dictionary_guide.md index 5ae81c3426d..7980395b1a9 100644 --- a/doc/development/snowplow/event_dictionary_guide.md +++ b/doc/development/snowplow/event_dictionary_guide.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index 88fb1d5cfe4..f8e37aee1e0 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -464,7 +464,7 @@ Page titles are hardcoded as `GitLab` for the same reason. #### Snowplow Inspector Chrome Extension -Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works in production, staging, and local development environments. +Snowplow Inspector Chrome Extension is a browser extension for testing frontend events. This works in production, staging, and local development environments. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a video tutorial, see the [Snowplow plugin walk through](https://www.youtube.com/watch?v=g4rqnIZ1Mb4). @@ -505,16 +505,22 @@ To install and run Snowplow Micro, complete these steps to modify the 1. Set the environment variable to tell the GDK to use Snowplow Micro in development. This overrides two `application_settings` options: - `snowplow_enabled` setting will instead return `true` from `Gitlab::Tracking.enabled?` - - `snowplow_collector_hostname` setting will instead always return `localhost:9090` (or whatever is set for `SNOWPLOW_MICRO_URI`) from `Gitlab::Tracking.collector_hostname`. + - `snowplow_collector_hostname` setting will instead always return `localhost:9090` (or whatever port is set for `snowplow_micro.port` GDK setting) from `Gitlab::Tracking.collector_hostname`. ```shell - export SNOWPLOW_MICRO_ENABLE=1 + gdk config set snowplow_micro.enabled true ``` - Optionally, you can set the URI for you Snowplow Micro instance as well (the default value is `http://localhost:9090`): + Optionally, you can set the port for you Snowplow Micro instance as well (the default value is `9090`): ```shell - export SNOWPLOW_MICRO_URI=https://127.0.0.1:8080 + gdk config set snowplow_micro.port 8080 + ``` + +1. Regenerate the project YAML config: + + ```shell + gdk reconfigure ``` 1. Restart GDK: diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index d6a7b900629..155ce87b8d9 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/infrastructure.md b/doc/development/snowplow/infrastructure.md index 28541874e98..758c850e89f 100644 --- a/doc/development/snowplow/infrastructure.md +++ b/doc/development/snowplow/infrastructure.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/review_guidelines.md b/doc/development/snowplow/review_guidelines.md index 0359636380b..673166452b7 100644 --- a/doc/development/snowplow/review_guidelines.md +++ b/doc/development/snowplow/review_guidelines.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/schemas.md b/doc/development/snowplow/schemas.md index 4066151600d..799f8335000 100644 --- a/doc/development/snowplow/schemas.md +++ b/doc/development/snowplow/schemas.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/snowplow/troubleshooting.md b/doc/development/snowplow/troubleshooting.md index 2a6db80a6f2..42a433e6a94 100644 --- a/doc/development/snowplow/troubleshooting.md +++ b/doc/development/snowplow/troubleshooting.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/development/stage_group_dashboards.md b/doc/development/stage_group_dashboards.md deleted file mode 100644 index 8e3e6982430..00000000000 --- a/doc/development/stage_group_dashboards.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: 'stage_group_observability/index.md' -remove_date: '2022-06-15' ---- - -This document was moved to [another location](stage_group_observability/index.md). - -<!-- This redirect file can be deleted after <2022-06-15>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index eda1c8c3d10..ea36214f6b7 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -155,16 +155,6 @@ FPROF=1 bin/rspec spec/[path]/[to]/[spec].rb FPROF=flamegraph bin/rspec spec/[path]/[to]/[spec].rb ``` -A common change is to use [`let_it_be`](#common-test-setup): - -```ruby -# Old -let(:project) { create(:project) } - -# New -let_it_be(:project) { create(:project) } -``` - A common cause of a large number of created factories is [factory cascades](https://github.com/test-prof/test-prof/blob/master/docs/profilers/factory_prof.md#factory-flamegraph), which result when factories create and recreate associations. They can be identified by a noticeable difference between `total time` and `top-level time` numbers: @@ -215,6 +205,56 @@ In this case, the `total time` and `top-level time` numbers match more closely: 8 8 0.0477s 0.0477s 0.0477s namespace ``` +##### Let's talk about `let` + +There are various ways to create objects and store them in variables in your tests. They are, from least efficient to most efficient: + +- `let!` creates the object before each example runs. It also creates a new object for every example. You should only use this option if you need to create a clean object before each example without explicitly referring to it. +- `let` lazily creates the object. It isn't created until the object is called. `let` is generally inefficient as it creates a new object for every example. `let` is fine for simple values. However, more efficient variants of `let` are best when dealing with database models such as factories. +- `let_it_be_with_refind` works similar to `let_it_be_with_reload`, but the [former calls `ActiveRecord::Base#find`](https://github.com/test-prof/test-prof/blob/936b29f87b36f88a134e064aa6d8ade143ae7a13/lib/test_prof/ext/active_record_refind.rb#L15) instead of `ActiveRecord::Base#reload`. `reload` is usually faster than `refind`. +- `let_it_be_with_reload` creates an object one time for all examples in the same context, but after each example, the database changes are rolled back, and `object.reload` will be called to restore the object to its original state. This means you can make changes to the object before or during an example. However, there are cases where [state leaks across other models](https://github.com/test-prof/test-prof/blob/master/docs/recipes/let_it_be.md#state-leakage-detection) can occur. In these cases, `let` may be an easier option, especially if only a few examples exist. +- `let_it_be` creates an immutable object one time for all of the examples in the same context. This is a great alternative to `let` and `let!` for objects that do not need to change from one example to another. Using `let_it_be` can dramatically speed up tests that create database models. See <https://github.com/test-prof/test-prof/blob/master/docs/recipes/let_it_be.md#let-it-be> for more details and examples. + +`let_it_be` is the most optimized option since it instantiates an object once and does not change it. If you find yourself needing `let` instead of `let_it_be`, try `let_it_be_with_reload`. + +```ruby +# Old +let(:project) { create(:project) } + +# New +let_it_be(:project) { create(:project) } + +# If you need to expect changes to the object in the test +let_it_be_with_reload(:project) { create(:project) } +``` + +Here is an example of when `let_it_be` cannot be used, but `let_it_be_with_reload` allows for more efficiency than `let`: + +```ruby +let_it_be(:user) { create(:user) } +let_it_be_with_reload(:project) { create(:project) } # The test will fail if `let_it_be` is used + +context 'with a developer' do + before do + project.add_developer(user) + end + + it 'project has an owner and a developer' do + expect(project.members.map(&:access_level)).to match_array([Gitlab::Access::OWNER, Gitlab::Access::DEVELOPER]) + end +end + +context 'with a maintainer' do + before do + project.add_maintainer(user) + end + + it 'project has an owner and a maintainer' do + expect(project.members.map(&:access_level)).to match_array([Gitlab::Access::OWNER, Gitlab::Access::MAINTAINER]) + end +end +``` + #### Stubbing methods within factories You should avoid using `allow(object).to receive(:method)` in factories, as this makes the factory unable to be used with `let_it_be`. diff --git a/doc/development/testing_guide/contract/consumer_tests.md b/doc/development/testing_guide/contract/consumer_tests.md index b4d6882a655..df7c9ee0abd 100644 --- a/doc/development/testing_guide/contract/consumer_tests.md +++ b/doc/development/testing_guide/contract/consumer_tests.md @@ -10,13 +10,15 @@ This tutorial guides you through writing a consumer test from scratch. To start, ## Create the skeleton -Start by creating the skeleton of a consumer test. Create a file under `spec/contracts/consumer/specs` called `discussions.spec.js`. +Start by creating the skeleton of a consumer test. Create a file under `spec/contracts/consumer/specs/project/merge_request` called `discussions.spec.js`. Then, populate it with the following function and parameters: - [`pactWith`](#the-pactwith-function) - [`PactOptions`](#the-pactoptions-parameter) - [`PactFn`](#the-pactfn-parameter) +To learn more about how the contract test directory is structured, see the contract testing [test suite folder structure](index.md#test-suite-folder-structure). + ### The `pactWith` function The Pact consumer test is defined through the `pactWith` function that takes `PactOptions` and the `PactFn`. @@ -36,15 +38,17 @@ const { pactWith } = require('jest-pact'); pactWith( { - consumer: 'Merge Request Page', + consumer: 'MergeRequest#show', provider: 'Merge Request Discussions Endpoint', log: '../logs/consumer.log', - dir: '../contracts', + dir: '../contracts/project/merge_request/show', }, PactFn ); ``` +To learn more about how to name the consumers and providers, see contract testing [naming conventions](index.md#naming-conventions). + ### The `PactFn` parameter The `PactFn` is where your tests are defined. This is where you set up the mock provider and where you can use the standard Jest methods like [`Jest.describe`](https://jestjs.io/docs/api#describename-fn), [`Jest.beforeEach`](https://jestjs.io/docs/api#beforeeachfn-timeout), and [`Jest.it`](https://jestjs.io/docs/api#testname-fn-timeout). For more information, see [https://jestjs.io/docs/api](https://jestjs.io/docs/api). @@ -54,20 +58,20 @@ const { pactWith } = require('jest-pact'); pactWith( { - consumer: 'Merge Request Page', + consumer: 'MergeRequest#show', provider: 'Merge Request Discussions Endpoint', log: '../logs/consumer.log', dir: '../contracts', }, (provider) => { - describe('Discussions Endpoint', () => { + describe('Merge Request Discussions Endpoint', () => { beforeEach(() => { - + }); it('return a successful body', () => { - + }); }); }, @@ -93,14 +97,14 @@ const { Matchers } = require('@pact-foundation/pact'); pactWith( { - consumer: 'Merge Request Page', + consumer: 'MergeRequest#show', provider: 'Merge Request Discussions Endpoint', log: '../logs/consumer.log', - dir: '../contracts', + dir: '../contracts/project/merge_request/show', }, (provider) => { - describe('Discussions Endpoint', () => { + describe('Merge Request Discussions Endpoint', () => { beforeEach(() => { const interaction = { state: 'a merge request with discussions exists', @@ -144,7 +148,7 @@ Notice how we use `Matchers` in the `body` of the expected response. This allows After the mock provider is set up, you can write the test. For this test, you make a request and expect a particular response. -First, set up the client that makes the API request. To do that, either create or find an existing file under `spec/contracts/consumer/endpoints` and add the following API request. +First, set up the client that makes the API request. To do that, create `spec/contracts/consumer/endpoints/project/merge_requests.js` and add the following API request. ```javascript const axios = require('axios'); @@ -169,18 +173,18 @@ After that's set up, import it to the test file and call it to make the request. const { pactWith } = require('jest-pact'); const { Matchers } = require('@pact-foundation/pact'); -const { getDiscussions } = require('../endpoints/merge_requests'); +const { getDiscussions } = require('../endpoints/project/merge_requests'); pactWith( { - consumer: 'Merge Request Page', + consumer: 'MergeRequest#show', provider: 'Merge Request Discussions Endpoint', log: '../logs/consumer.log', - dir: '../contracts', + dir: '../contracts/project/merge_request/show', }, (provider) => { - describe('Discussions Endpoint', () => { + describe('Merge Request Discussions Endpoint', () => { beforeEach(() => { const interaction = { state: 'a merge request with discussions exists', @@ -230,7 +234,7 @@ There we have it! The consumer test is now set up. You can now try [running this As you may have noticed, the request and response definitions can get large. This results in the test being difficult to read, with a lot of scrolling to find what you want. You can make the test easier to read by extracting these out to a `fixture`. -Create a file under `spec/contracts/consumer/fixtures` called `discussions.fixture.js`. You place the `request` and `response` definitions here. +Create a file under `spec/contracts/consumer/fixtures/project/merge_request` called `discussions.fixture.js` where you will place the `request` and `response` definitions. ```javascript const { Matchers } = require('@pact-foundation/pact'); @@ -274,18 +278,18 @@ With all of that moved to the `fixture`, you can simplify the test to the follow const { pactWith } = require('jest-pact'); const { Discussions } = require('../fixtures/discussions.fixture'); -const { getDiscussions } = require('../endpoints/merge_requests'); +const { getDiscussions } = require('../endpoints/project/merge_requests'); pactWith( { - consumer: 'Merge Request Page', + consumer: 'MergeRequest#show', provider: 'Merge Request Discussions Endpoint', log: '../logs/consumer.log', - dir: '../contracts', + dir: '../contracts/project/merge_request/show', }, (provider) => { - describe('Discussions Endpoint', () => { + describe('Merge Request Discussions Endpoint', () => { beforeEach(() => { const interaction = { state: 'a merge request with discussions exists', diff --git a/doc/development/testing_guide/contract/index.md b/doc/development/testing_guide/contract/index.md index 6556bd85624..8e12eea2874 100644 --- a/doc/development/testing_guide/contract/index.md +++ b/doc/development/testing_guide/contract/index.md @@ -37,3 +37,42 @@ rake contracts:mr:pact:verify:discussions # Verify provider against the rake contracts:mr:pact:verify:metadata # Verify provider against the consumer pacts for metadata rake contracts:mr:test:merge_request[contract_mr] # Run all merge request contract tests ``` + +## Test suite folder structure and naming conventions + +To keep the consumer and provider test suite organized and maintainable, it's important that tests are organized, also that consumers and providers are named consistently. Therefore, it's important to adhere to the following conventions. + +### Test suite folder structure + +Having an organized and sensible folder structure for the test suite makes it easier to find relevant files when reviewing, debugging, or introducing tests. + +#### Consumer tests + +The consumer tests are grouped according to the different pages in the application. Each file contains various types of requests found in a page. As such, the consumer test files are named using the Rails standards of how pages are referenced. For example, the project pipelines page would be the `Project::Pipeline#index` page so the equivalent consumer test would be located in `consumer/specs/project/pipelines/index.spec.js`. + +When defining the location to output the contract generated by the test, we want to follow the same file structure which would be `contracts/project/pipelines/` for this example. This is the structure in `consumer/endpoints` and `consumer/fixtures` as well. + +#### Provider tests + +The provider tests are grouped similarly to our controllers. Each of these tests contains various tests for an API endpoint. For example, the API endpoint to get a list of pipelines for a project would be located in `provider/pact_helpers/project/pipelines/get_list_project_pipelines_helper.rb`. The provider states are structured the same way. + +### Naming conventions + +When writing the consumer and provider tests, there are parts where a name is required for the consumer and provider. Since there are no restrictions imposed by Pact on how these should be named, a naming convention is important to keep it easy for us to figure out which consumer and provider tests are involved during debugging. Pact also uses the consumer and provider names to generate the generated contracts in the `#{consumer_name}-#{provider_name}` format. + +#### Consumer naming + +As mentioned in the [folder structure section](#consumer-tests), consumer tests are grouped according to the different pages in the application. As such, consumer names should follow the same naming format using the Rails standard. For example, the consumer test for `Project::Pipeline#index` would be `ProjectPipeline#index` as the consumer name. Since Pact uses this name to name the contracts it generates, the colons (`::`) are dropped as colons are not valid characters in file names. + +#### Provider naming + +These are the API endpoints that provides the data to the consumer so they are simply named according to the API endpoint they pertain to. Be mindful that this name is as descriptive as possible. For example, if we're writing a test for the `GET /groups/:id/projects` endpoint, we don't want to simply name it "Projects endpoint" as there is a `GET /projects` endpoint as well that also fetches a list of projects the user has access to across all of GitLab. An easy way to name them is by checking out our [API documentation](../../../api/api_resources.md) and naming it the same way it is named in there. So the [`GET /groups/:id/projects`](../../../api/groups.md#list-a-groups-projects) would be called `List a group’s projects` and [`GET /projects`](../../../api/projects.md#list-all-projects) would be called `List all projects`. Subsequently, the test files are named `list_a_groups_projects_helper.rb` and `list_all_projects_helper.rb` respectively. + +There are some cases where the provider being tested may not be documented so, in those cases, fall back to choosing a name that is as descriptive as possible to ensure it's easy to tell what the provider is for. + +#### Conventions summary + +| Tests | Folder structure | Naming convention | +| ----- | ---------------- | ----------------- | +| Consumer Test | Follows the Rails reference standards. For example, `Project::Pipeline#index` would be `consumer/specs/project/pipelines/index.spec.js` | Follows the Rails naming standard. For example, `Project::Pipeline#index` would be `ProjectPipeline#index` | +| Provider Test | Grouped like the Rails controllers. For example, [`List project pipelines` API endpoint](../../../api/pipelines.md#list-project-pipelines) would be `provider/pact_helpers/project/pipelines/provider/pact_helpers/project/pipelines/get_list_project_pipelines_helper.rb` | Follows the API documentation naming scheme. For example, [`GET /projects/:id/pipelines`](../../../api/pipelines.md#list-project-pipelines) would be called `List project pipelines`. | diff --git a/doc/development/testing_guide/contract/provider_tests.md b/doc/development/testing_guide/contract/provider_tests.md index 0da5bcb4aef..92ac4c4ed71 100644 --- a/doc/development/testing_guide/contract/provider_tests.md +++ b/doc/development/testing_guide/contract/provider_tests.md @@ -6,23 +6,25 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Writing provider tests -This tutorial guides you through writing a provider test from scratch. It is a continuation of the [consumer test tutorial](consumer_tests.md). To start, the provider tests are written using [`pact-ruby`](https://github.com/pact-foundation/pact-ruby). In this tutorial, you write a provider test that addresses the contract generated by `discussions.spec.js`. +This tutorial guides you through writing a provider test from scratch. It is a continuation of the [consumer test tutorial](consumer_tests.md). To start, the provider tests are written using [`pact-ruby`](https://github.com/pact-foundation/pact-ruby). In this tutorial, you write a provider test that addresses the contract generated by `discussions.spec.js`. As Pact is a consumer-driven testing tool, this tutorial assumes that there is an existing consumer test that had already generated a contract for us to work with. ## Create the skeleton -Provider tests are quite simple. The goal is to set up the test data and then link that with the corresponding contract. Start by creating a file called `discussions_helper.rb` under `spec/contracts/provider/specs`. Note that the files are called `helpers` to match how they are called by Pact in the Rake tasks, which are set up at the end of this tutorial. +Provider tests are quite simple. The goal is to set up the test data and then link that with the corresponding contract. Start by creating a file called `discussions_helper.rb` under `spec/contracts/provider/pact_helpers/project/merge_request`. Note that the files are called `helpers` to match how they are called by Pact in the Rake tasks, which are set up at the end of this tutorial. + +To learn more about how the contract test directory is structured, see the contract testing [test suite folder structure](index.md#test-suite-folder-structure). ### The `service_provider` block The `service_provider` block is where the provider test is defined. For this block, put in a description of the service provider. Name it exactly as it is called in the contracts that are derived from the consumer tests. ```ruby -require_relative '../spec_helper' +require_relative '../../../spec_helper' module Provider module DiscussionsHelper Pact.service_provider 'Merge Request Discussions Endpoint' do - + end end end @@ -33,33 +35,35 @@ end The `honours_pact_with` block describes which consumer this provider test is addressing. Similar to the `service_provider` block, name this exactly the same as it's called in the contracts that are derived from the consumer tests. ```ruby -require_relative '../spec_helper' +require_relative '../../../spec_helper' module Provider module DiscussionsHelper Pact.service_provider 'Merge Request Discussions Endpoint' do - honours_pact_with 'Merge Request Page' do - + honours_pact_with 'MergeRequest#show' do + end end end end ``` +To learn more about how to name the consumers and providers, see contract testing [naming conventions](index.md#naming-conventions). + ## Configure the test app For the provider tests to verify the contracts, you must hook it up to a test app that makes the actual request and return a response to verify against the contract. To do this, configure the `app` the test uses as `Environment::Test.app`, which is defined in [`spec/contracts/provider/environments/test.rb`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/provider/environments/test.rb). ```ruby -require_relative '../spec_helper' +require_relative '../../../spec_helper' module Provider module DiscussionsHelper Pact.service_provider 'Merge Request Discussions Endpoint' do app { Environment::Test.app } - - honours_pact_with 'Merge Request Page' do - + + honours_pact_with 'MergeRequest#show' do + end end end @@ -71,15 +75,15 @@ end Now that the test app is configured, all that is left is to define which contract this provider test is verifying. To do this, set the `pact_uri`. ```ruby -require_relative '../spec_helper' +require_relative '../../../spec_helper' module Provider module DiscussionsHelper Pact.service_provider 'Merge Request Discussions Endpoint' do app { Environment::Test.app } - - honours_pact_with 'Merge Request Page' do - pact_uri '../contracts/merge_request_page-merge_request_discussions_endpoint.json' + + honours_pact_with 'MergeRequest#show' do + pact_uri '../contracts/project/merge_request/show/mergerequest#show-merge_request_discussions_endpoint.json' end end end @@ -95,8 +99,8 @@ Under the `contracts:mr` namespace, introduce the Rake task to run this new test ```ruby Pact::VerificationTask.new(:discussions) do |pact| pact.uri( - "#{contracts}/contracts/merge_request_page-merge_request_discussions_endpoint.json", - pact_helper: "#{provider}/specs/discussions_helper.rb" + "#{contracts}/contracts/project/merge_request/show/merge_request#show-merge_request_discussions_endpoint.json", + pact_helper: "#{provider}/pact_helpers/project/merge_request/discussions_helper.rb" ) end ``` @@ -109,7 +113,7 @@ As the last step, create the test data that allows the provider test to return t You can read more about [provider states](https://docs.pact.io/implementation_guides/ruby/provider_states). We can do global provider states but for this tutorial, the provider state is for one specific `state`. -To create the test data, create `discussions_state.rb` under `spec/contracts/provider/states`. As a quick aside, make sure to also import this state file in the `discussions_helper.rb` file. +To create the test data, create `discussions_state.rb` under `spec/contracts/provider/states/project/merge_request`. Be sure to also import this state file in the `discussions_helper.rb` file. ### Default user in `spec/contracts/provider/spec_helper.rb` @@ -118,10 +122,13 @@ Before you create the test data, note that a default user is created in the [`sp ```ruby RSpec.configure do |config| config.include Devise::Test::IntegrationHelpers + config.include FactoryBot::Syntax::Methods + config.before do - user = FactoryBot.create(:user, name: "Contract Test").tap do |user| + user = create(:user, name: Provider::UsersHelper::CONTRACT_USER_NAME).tap do |user| user.current_sign_in_at = Time.current end + sign_in user end end @@ -134,7 +141,7 @@ Any further modifications to the user that's needed can be done through the indi In the state file, you must define which consumer this provider state is for. You can do that with `provider_states_for`. Make sure that the `name` provided matches the name defined for the consumer. ```ruby -Pact.provider_states_for 'Merge Request Page' do +Pact.provider_states_for 'MergeRequest#show' do end ``` @@ -143,7 +150,7 @@ end In the `provider_states_for` block, you then define the state the test data is for. These states are also defined in the consumer test. In this case, there is a `'a merge request with discussions exists'` state. ```ruby -Pact.provider_states_for "Merge Request Page" do +Pact.provider_states_for "MergeRequest#show" do provider_state "a merge request with discussions exists" do end @@ -155,7 +162,7 @@ end This is where you define the test data creation steps. Use `FactoryBot` to create the data. As you create the test data, you can keep [running the provider test](index.md#run-the-provider-tests) to check on the status of the test and figure out what else is missing in your data setup. ```ruby -Pact.provider_states_for "Merge Request Page" do +Pact.provider_states_for "MergeRequest#show" do provider_state "a merge request with discussions exists" do set_up do user = User.find_by(name: Provider::UsersHelper::CONTRACT_USER_NAME) @@ -172,6 +179,28 @@ Pact.provider_states_for "Merge Request Page" do end ``` -Note the `Provider::UsersHelper::CONTRACT_USER_NAME` here to fetch a user is a user that is from the [`spec_helper`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/spec/contracts/provider/spec_helper.rb) that sets up a user before any of these tests run. +## Using the test data + +Now that the provider state file is created, you need to import the state file to the provider test. + +```ruby +# frozen_string_literal: true + +require_relative '../../../spec_helper' +require_relative '../../../states/project/merge_request/discussions_state' + +module Provider + module DiscussionsHelper + Pact.service_provider "/merge_request/discussions" do + app { Environments::Test.app } + + honours_pact_with 'Merge Request#show' do + pact_uri '../contracts/project/merge_request/show/merge_request#show-merge_request_discussions_endpoint.json' + end + end + end +end + +``` -And with that, the provider tests for `discussion_helper.rb` should now pass with this. +And there we have it. The provider test for `discussions_helper.rb` should now pass with this. diff --git a/doc/development/testing_guide/end_to_end/best_practices.md b/doc/development/testing_guide/end_to_end/best_practices.md index 85f8beeacad..00b843ffdbe 100644 --- a/doc/development/testing_guide/end_to_end/best_practices.md +++ b/doc/development/testing_guide/end_to_end/best_practices.md @@ -220,7 +220,7 @@ For example, if you encapsulate some actions and expectations in a private metho it "has Owner role with Owner permissions" do Page::Dashboard::Projects.perform do |projects| projects.filter_by_name(project.name) - + expect(projects).to have_project_with_access_role(project.name, 'Owner') end diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index 9730115fd9f..06359d612ad 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -174,7 +174,7 @@ See [Review Apps](../review_apps.md) for more details about Review Apps. To run tests in parallel on CI, the [Knapsack](https://github.com/KnapsackPro/knapsack) gem is used. Knapsack reports are generated automatically and stored in the `GCS` bucket -`knapsack-reports` in the `gitlab-qa-resources` project. The [`KnapsackReport`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/tools/knapsack_report.rb) +`knapsack-reports` in the `gitlab-qa-resources` project. The [`KnapsackReport`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/qa/qa/support/knapsack_report.rb) helper handles automated report generation and upload. ## Test metrics diff --git a/doc/development/testing_guide/end_to_end/resources.md b/doc/development/testing_guide/end_to_end/resources.md index dacc428aec6..a519d1ecb47 100644 --- a/doc/development/testing_guide/end_to_end/resources.md +++ b/doc/development/testing_guide/end_to_end/resources.md @@ -568,6 +568,16 @@ def unique_identifiers end ``` +### Resources cleanup + +We have a mechanism to [collect](https://gitlab.com/gitlab-org/gitlab/-/blob/44345381e89d6bbd440f7b4c680d03e8b75b86de/qa/qa/tools/test_resource_data_processor.rb#L32) +all resources created during test executions, and another to [handle](https://gitlab.com/gitlab-org/gitlab/-/blob/44345381e89d6bbd440f7b4c680d03e8b75b86de/qa/qa/tools/test_resources_handler.rb#L44) +these resources. On [dotcom environments](https://about.gitlab.com/handbook/engineering/infrastructure/environments/#environments), after a test suite finishes in the [QA pipelines](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/debugging-qa-test-failures/#scheduled-qa-test-pipelines), resources from all passing test are +automatically deleted in the same pipeline run. Resources from all failed tests are reserved for investigation, +and won't be deleted until the following Saturday by a scheduled pipeline. When introducing new resources, please +also make sure to add any resource that cannot be deleted to the [IGNORED_RESOURCES](https://gitlab.com/gitlab-org/gitlab/-/blob/44345381e89d6bbd440f7b4c680d03e8b75b86de/qa/qa/tools/test_resources_handler.rb#L29) +list. + ## Where to ask for help? If you need more information, ask for help on `#quality` channel on Slack diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index f1083c23406..532bb9fcdef 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -45,7 +45,7 @@ Maintainers can elect to use the [process for merging during broken `master`](ht On every [pipeline](https://gitlab.com/gitlab-org/gitlab/pipelines/125315730) in the `qa` stage, the `review-performance` job is automatically started: this job does basic browser performance testing using a -[Sitespeed.io Container](../../user/project/merge_requests/browser_performance_testing.md). +[Sitespeed.io Container](../../ci/testing/browser_performance_testing.md). ## Sample Data for Review Apps diff --git a/doc/development/uploads.md b/doc/development/uploads.md deleted file mode 100644 index 1860f898a26..00000000000 --- a/doc/development/uploads.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'uploads/index.md' -remove_date: '2022-04-30' ---- - -This document was moved to [another location](uploads/index.md). - -<!-- This redirect file can be deleted after 2022-04-30. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md index aef85107cd9..79262e2d0dc 100644 --- a/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md +++ b/doc/development/value_stream_analytics/value_stream_analytics_aggregated_backend.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Aggregated Value Stream Analytics -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335391) in GitLab 14.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335391) in GitLab 14.7. DISCLAIMER: This page contains information related to upcoming products, features, and functionality. diff --git a/doc/development/workhorse/new_features.md b/doc/development/workhorse/new_features.md index 3ad15c1de16..5c00903497a 100644 --- a/doc/development/workhorse/new_features.md +++ b/doc/development/workhorse/new_features.md @@ -74,5 +74,5 @@ The Workhorse maintainers can help you assess the situation. - In 2020, `@nolith` presented the talk ["Speed up the monolith. Building a smart reverse proxy in Go"](https://archive.fosdem.org/2020/schedule/event/speedupmonolith/) at FOSDEM. The talk includes more details on the history of Workhorse and the NFS removal. -- The [uploads development documentation](../uploads.md) contains the most common +- The [uploads development documentation](../uploads/index.md) contains the most common use cases for adding a new type of upload. diff --git a/doc/gitlab-basics/add-file.md b/doc/gitlab-basics/add-file.md index 6dd0c608983..af736c11d59 100644 --- a/doc/gitlab-basics/add-file.md +++ b/doc/gitlab-basics/add-file.md @@ -29,7 +29,7 @@ to the desired destination: cd <destination folder> ``` -[Create a new branch](create-branch.md) to add your file into. Submitting changes directly +[Create a new branch](../tutorials/make_your_first_git_commit.md#create-a-branch-and-make-changes) to add your file into. Submitting changes directly to the default branch should be avoided unless your project is very small and you're the only person working on it. diff --git a/doc/gitlab-basics/create-branch.md b/doc/gitlab-basics/create-branch.md deleted file mode 100644 index d22ce12f9cd..00000000000 --- a/doc/gitlab-basics/create-branch.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../tutorials/make_your_first_git_commit.md' -remove_date: '2022-06-26' ---- - -This document was moved to [another location](../tutorials/make_your_first_git_commit.md). - -<!-- This redirect file can be deleted after <YYYY-MM-DD>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html -->
\ No newline at end of file diff --git a/doc/install/aws/eks_clusters_aws.md b/doc/install/aws/eks_clusters_aws.md index 342740994a1..c939bac888c 100644 --- a/doc/install/aws/eks_clusters_aws.md +++ b/doc/install/aws/eks_clusters_aws.md @@ -48,4 +48,4 @@ Read how to [use the GitLab Runner Helm Chart](https://docs.gitlab.com/runner/in ## Runner Cache -Since the EKS Quick Start provides for EFS provisioning, the best approach is to use EFS for runner caching. Eventually we will publish information on using an S3 bucket for runner caching here. +Because the EKS Quick Start provides for EFS provisioning, the best approach is to use EFS for runner caching. Eventually we will publish information on using an S3 bucket for runner caching here. diff --git a/doc/install/aws/gitlab_sre_for_aws.md b/doc/install/aws/gitlab_sre_for_aws.md index cb7703835fa..2e7de6cf2d4 100644 --- a/doc/install/aws/gitlab_sre_for_aws.md +++ b/doc/install/aws/gitlab_sre_for_aws.md @@ -23,15 +23,15 @@ If you would like to understand the underlying rationale on why GitLab had to in ### Gitaly and Praefect elections -As part of Gitaly cluster consistency, Praefect nodes will occasionally need to vote on what data copy is the most accurate. This requires an uneven number of Praefect nodes to avoid stalemates. This means that for HA, Gitaly and Praefect require a minimum of three nodes. +As part of Gitaly cluster consistency, Praefect nodes must occasionally vote on what data copy is the most accurate. This requires an uneven number of Praefect nodes to avoid stalemates. This means that for HA, Gitaly and Praefect require a minimum of three nodes. ### Gitaly performance monitoring -Complete performance metrics should be collected for Gitaly instances for identification of bottlenecks, as they could have to do with disk IO, network IO or memory. +Complete performance metrics should be collected for Gitaly instances for identification of bottlenecks, as they could have to do with disk IO, network IO, or memory. ### Gitaly performance guidelines -Gitaly functions as the primary Git Repository Storage in GitLab. However, it's not simply a streaming file server. It also does a lot of demanding computing work, such as preparing and caching Git pack files which informs some of the performance recommendations below. +Gitaly functions as the primary Git Repository Storage in GitLab. However, it's not a streaming file server. It also does a lot of demanding computing work, such as preparing and caching Git pack files which informs some of the performance recommendations below. NOTE: All recommendations are for production configurations, including performance testing. For test configurations, like training or functional testing, you can use less expensive options. However, you should adjust or rebuild if performance is an issue. @@ -45,7 +45,7 @@ All recommendations are for production configurations, including performance tes #### CPU and memory recommendations -- The general GitLab Gitaly node recommendations for CPU and Memory assume relatively even loading across repositories. GPT testing of any non-characteristic repositories and/or SRE monitoring of Gitaly metrics may inform when to choose memory and/or CPU higher than general recommendations. +- The general GitLab Gitaly node recommendations for CPU and Memory assume relatively even loading across repositories. GitLab Performance Tool (GPT) testing of any non-characteristic repositories and/or SRE monitoring of Gitaly metrics may inform when to choose memory and/or CPU higher than general recommendations. **To accommodate:** @@ -54,9 +54,9 @@ All recommendations are for production configurations, including performance tes #### Disk I/O recommendations -- Use only SSD storage and the [class of EBS storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html) that suites your durability and speed requirements. +- Use only SSD storage and the [class of Elastic Block Store (EBS) storage](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-volume-types.html) that suites your durability and speed requirements. - When not using provisioned EBS IO, EBS volume size determines the I/O level, so provisioning volumes that are much larger than needed can be the least expensive way to improve EBS IO. -- If Gitaly performance monitoring shows signs of disk stress then one of the provisioned IOPs levels can be chosen. Note that EBS IOPs levels also have enhanced durability which may be appealing for some implementations aside from performance considerations. +- If Gitaly performance monitoring shows signs of disk stress then one of the provisioned IOPs levels can be chosen. EBS IOPs levels also have enhanced durability which may be appealing for some implementations aside from performance considerations. **To accommodate:** @@ -66,7 +66,7 @@ All recommendations are for production configurations, including performance tes #### Network I/O recommendations -- Use only instance types [from the list of ones that support ENA advanced networking]( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#instance-type-summary-table) to ensure that cluster replication latency is not due to instance level network I/O bottlenecks. +- Use only instance types [from the list of ones that support Elastic Network Adapter (ENA) advanced networking](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-types.html#instance-type-summary-table) to ensure that cluster replication latency is not due to instance level network I/O bottlenecks. - Choose instances with sizes with more than 10 Gbps - but only if needed and only when having proven a node level network bottleneck with monitoring and/or stress testing. **To accommodate:** @@ -93,4 +93,4 @@ The [AWS GitLab Cloud Native Hybrid on EKS Quick Start](gitlab_hybrid_on_aws.md# ### Gitaly long term management -Gitaly node disk sizes will need to be monitored and increased to accommodate Git repository growth and Gitaly temporary and caching storage needs. The storage configuration on all nodes should be kept identical. +Gitaly node disk sizes must be monitored and increased to accommodate Git repository growth and Gitaly temporary and caching storage needs. The storage configuration on all nodes should be kept identical. diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index f54bc979152..1227dd43369 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -11,7 +11,7 @@ type: index GitLab [Reference Architectures](../../administration/reference_architectures/index.md) give qualified and tested guidance on the recommended ways GitLab can be configured to meet the performance requirements of various workloads. Reference Architectures are purpose-designed to be non-implementation specific so they can be extrapolated to as many environments as possible. This generally means they have a highly-granular "machine" to "server role" specification and focus on system elements that impact performance. This is what enables Reference Architectures to be adaptable to the broadest number of supported implementations. -Implementation patterns are built on the foundational information and testing done for Reference Architectures and allow architects and implementers at GitLab, GitLab Customers, and GitLab Partners to build out deployments with less experimentation and a higher degree of confidence that the results will perform as expected. A more thorough discussion of implementation patterns is below in [Additional details on implementation patterns](#additional-details-on-implementation-patterns). +Implementation patterns are built on the foundational information and testing done for Reference Architectures and allow architects and implementers at GitLab, GitLab Customers, and GitLab Partners to build out deployments with less experimentation and a higher degree of confidence that the results perform as expected. A more thorough discussion of implementation patterns is below in [Additional details on implementation patterns](#additional-details-on-implementation-patterns). ## AWS Implementation patterns information @@ -19,7 +19,7 @@ The following are the currently available implementation patterns for GitLab whe ### GitLab Site Reliability Engineering (SRE) for AWS -[GitLab Site Reliability Engineering (SRE) for AWS](gitlab_sre_for_aws.md) - information for planning, implementing, upgrading and long term management of GitLab instances and runners on AWS. +[GitLab Site Reliability Engineering (SRE) for AWS](gitlab_sre_for_aws.md) - information for planning, implementing, upgrading, and long term management of GitLab instances and runners on AWS. ### Patterns to Install GitLab Cloud Native Hybrid on AWS EKS (HA) @@ -33,7 +33,7 @@ The following are the currently available implementation patterns for GitLab whe [EKS Cluster Provisioning Patterns](eks_clusters_aws.md) - considerations for setting up EKS cluster for runners and for integrating. -### Patterns for Scaling HA GitLab Runner on AWS EC2 ASG +### Patterns for Scaling HA GitLab Runner on AWS EC2 Auto Scaling group (ASG) The following repository is self-contained in regard to enabling this pattern: [GitLab HA Scaling Runner Vending Machine for AWS EC2 ASG](https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/). The [feature list for this implementation pattern](https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/-/blob/main/FEATURES.md) is good to review to understand the complete value it can deliver. @@ -47,23 +47,23 @@ The following repository is self-contained in regard to enabling this pattern: [ ## AWS known issues list -Known issues are gathered from within GitLab and from customer reported issues. Customers successfully implement GitLab with a variety of "as a Service" components that GitLab has not specifically been designed for, nor has ongoing testing for. While GitLab does take partner technologies very seriously, the highlighting of known issues here is a convenience for implementers and it does not imply that GitLab has targeted compatibility with, nor carries any type of guarantee of running on the partner technology where the issues occur. Please consult individual issues to understand GitLabs stance and plans on any given known issue. +Known issues are gathered from within GitLab and from customer reported issues. Customers successfully implement GitLab with a variety of "as a Service" components that GitLab has not specifically been designed for, nor has ongoing testing for. While GitLab does take partner technologies very seriously, the highlighting of known issues here is a convenience for implementers and it does not imply that GitLab has targeted compatibility with, nor carries any type of guarantee of running on the partner technology where the issues occur. Consult individual issues to understand the GitLab stance and plans on any given known issue. See the [GitLab AWS known issues list](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues?label_name%5B%5D=AWS+Known+Issue) for a complete list. ## Official GitLab releases as AMIs -GitLab produces AMI images during the regular release process. The AMIs can be used for single instance GitLab installation or, by configuring `/etc/gitlab/gitlab.rb`, can be specialized for specific GitLab service roles (for example a Gitaly server). Older releases remain available and can be used to migrate an older GitLab server to AWS. +GitLab produces Amazon Machine Images (AMI) during the regular release process. The AMIs can be used for single instance GitLab installation or, by configuring `/etc/gitlab/gitlab.rb`, can be specialized for specific GitLab service roles (for example a Gitaly server). Older releases remain available and can be used to migrate an older GitLab server to AWS. Currently the Amazon AMI uses the Amazon prepared Ubuntu AMI (x86 and ARM are available) as its starting point. NOTE: -When deploying a GitLab instance using the official AMI, the root password to the instance will be the EC2 **Instance** ID (NOT the AMI ID). This way of setting the root account password is specific to official GitLab published AMIs ONLY. +When deploying a GitLab instance using the official AMI, the root password to the instance is the EC2 **Instance** ID (not the AMI ID). This way of setting the root account password is specific to official GitLab published AMIs ONLY. -Instances running on Community Edition (CE) require a migration to Enterprise Edition (EE) in order to subscribe to the GitLab Premium or Ultimate plan. If you want to pursue a subscription, using the Free-forever plan of Enterprise Edition is the least disruptive method. +Instances running on Community Edition (CE) require a migration to Enterprise Edition (EE) to subscribe to the GitLab Premium or Ultimate plan. If you want to pursue a subscription, using the Free-forever plan of Enterprise Edition is the least disruptive method. NOTE: -Since any given GitLab upgrade might involve data disk updates or database schema upgrades, simply swapping out the AMI is not sufficient for taking upgrades. +Because any given GitLab upgrade might involve data disk updates or database schema upgrades, swapping out the AMI is not sufficient for taking upgrades. 1. Log in to the AWS Web Console, so that clicking the links in the following step take you directly to the AMI list. 1. Pick the edition you want: @@ -72,9 +72,9 @@ Since any given GitLab upgrade might involve data disk updates or database schem - [GitLab Community Edition](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;ownerAlias=782774275127;search=GitLab%20CE;sort=desc:name): The open source version of GitLab. - [GitLab Premium or Ultimate Marketplace (Prelicensed)](https://console.aws.amazon.com/ec2/v2/home?region=us-east-1#Images:visibility=public-images;source=Marketplace;search=GitLab%20EE;sort=desc:name): 5 user license built into per-minute billing. -1. AMI IDs are unique per region, so once you've loaded one of the above, select the desired target region in the upper right of the console to see the appropriate AMIs. +1. AMI IDs are unique per region, so after you've loaded one of the above, select the desired target region in the upper right of the console to see the appropriate AMIs. 1. After the console is loaded, you can add additional search criteria to narrow further. For instance, type `13.` to find only 13.x versions. -1. To launch an EC2 Machine with one of the listed AMIs, check the box at the start of the relevant row, and select the "Launch" button near the top of left of the page. +1. To launch an EC2 Machine with one of the listed AMIs, check the box at the start of the relevant row, and select **Launch** near the top of left of the page. NOTE: If you are trying to restore from an older version of GitLab while moving to AWS, find the @@ -88,7 +88,7 @@ GitLab implementation patterns build upon [GitLab Reference Architectures](../.. Testing-backed architectural qualification is a fundamental concept behind implementation patterns: -- Implementation patterns maintain GitLab Reference Architecture compliance and provide [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) (gpt) reports to demonstrate adherence to them. +- Implementation patterns maintain GitLab Reference Architecture compliance and provide [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance) (GPT) reports to demonstrate adherence to them. - Implementation patterns may be qualified by and/or contributed to by the technology vendor. For instance, an implementation pattern for AWS may be officially reviewed by AWS. - Implementation patterns may specify and test Cloud Platform PaaS services for suitability for GitLab. This testing can be coordinated and help qualify these technologies for Reference Architectures. For instance, qualifying compatibility with and availability of runtime versions of top level PaaS such as those for PostgreSQL and Redis. - Implementation patterns can provided qualified testing for platform limitations, for example, ensuring Gitaly Cluster can work correctly on specific Cloud Platform availability zone latency and throughput characteristics or qualifying what levels of available platform partner local disk performance is workable for Gitaly server to operate with integrity. diff --git a/doc/install/aws/manual_install_aws.md b/doc/install/aws/manual_install_aws.md index 25973220170..7494dc1e95e 100644 --- a/doc/install/aws/manual_install_aws.md +++ b/doc/install/aws/manual_install_aws.md @@ -32,23 +32,23 @@ For the Cloud Native Hybrid architectures there are two Infrastructure as Code o ## Introduction -For the most part, we'll make use of Omnibus GitLab in our setup, but we'll also leverage native AWS services. Instead of using the Omnibus bundled PostgreSQL and Redis, we will use Amazon RDS and ElastiCache. +For the most part, we make use of Omnibus GitLab in our setup, but we also leverage native AWS services. Instead of using the Omnibus bundled PostgreSQL and Redis, we use Amazon RDS and ElastiCache. -In this guide, we'll go through a multi-node setup where we'll start by +In this guide, we go through a multi-node setup where we start by configuring our Virtual Private Cloud and subnets to later integrate services such as RDS for our database server and ElastiCache as a Redis -cluster to finally manage them within an auto scaling group with custom +cluster to finally manage them in an auto scaling group with custom scaling policies. ## Requirements -In addition to having a basic familiarity with [AWS](https://docs.aws.amazon.com/) and [Amazon EC2](https://docs.aws.amazon.com/ec2/), you will need: +In addition to having a basic familiarity with [AWS](https://docs.aws.amazon.com/) and [Amazon EC2](https://docs.aws.amazon.com/ec2/), you need: - [An AWS account](https://console.aws.amazon.com/console/home) - [To create or upload an SSH key](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html) to connect to the instance via SSH - A domain name for the GitLab instance -- An SSL/TLS certificate to secure your domain. If you do not already own one, you can provision a free public SSL/TLS certificate through [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/)(ACM) for use with the [Elastic Load Balancer](#load-balancer) we'll create. +- An SSL/TLS certificate to secure your domain. If you do not already own one, you can provision a free public SSL/TLS certificate through [AWS Certificate Manager](https://aws.amazon.com/certificate-manager/)(ACM) for use with the [Elastic Load Balancer](#load-balancer) we create. NOTE: It can take a few hours to validate a certificate provisioned through ACM. To avoid delays later, request your certificate as soon as possible. @@ -79,11 +79,11 @@ GitLab uses the following AWS services, with links to pricing information: ## Create an IAM EC2 instance role and profile -As we'll be using [Amazon S3 object storage](#amazon-s3-object-storage), our EC2 instances need to have read, write, and list permissions for our S3 buckets. To avoid embedding AWS keys in our GitLab configuration, we'll make use of an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) to allow our GitLab instance with this access. We'll need to create an IAM policy to attach to our IAM role: +As we are using [Amazon S3 object storage](#amazon-s3-object-storage), our EC2 instances must have read, write, and list permissions for our S3 buckets. To avoid embedding AWS keys in our GitLab configuration, we make use of an [IAM Role](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles.html) to allow our GitLab instance with this access. We must create an IAM policy to attach to our IAM role: ### Create an IAM Policy -1. Navigate to the IAM dashboard and select **Policies** in the left menu. +1. Go to the IAM dashboard and select **Policies** in the left menu. 1. Select **Create policy**, select the `JSON` tab, and add a policy. We want to [follow security best practices and grant _least privilege_](https://docs.aws.amazon.com/IAM/latest/UserGuide/best-practices.html#grant-least-privilege), giving our role only the permissions needed to perform the required actions. 1. Assuming you prefix the S3 bucket names with `gl-` as shown in the diagram, add the following policy: @@ -114,7 +114,7 @@ As we'll be using [Amazon S3 object storage](#amazon-s3-object-storage), our EC2 } ``` -1. Select **Review policy**, give your policy a name (we'll use `gl-s3-policy`), and select **Create policy**. +1. Select **Review policy**, give your policy a name (we use `gl-s3-policy`), and select **Create policy**. ### Create an IAM Role @@ -124,20 +124,20 @@ As we'll be using [Amazon S3 object storage](#amazon-s3-object-storage), our EC2 **Next: Permissions**. 1. In the policy filter, search for the `gl-s3-policy` we created above, select it, and select **Tags**. 1. Add tags if needed and select **Review**. -1. Give the role a name (we'll use `GitLabS3Access`) and select **Create Role**. +1. Give the role a name (we use `GitLabS3Access`) and select **Create Role**. -We'll use this role when we [create a launch configuration](#create-a-launch-configuration) later on. +We use this role when we [create a launch configuration](#create-a-launch-configuration) later on. ## Configuring the network -We'll start by creating a VPC for our GitLab cloud infrastructure, then +We start by creating a VPC for our GitLab cloud infrastructure, then we can create subnets to have public and private instances in at least -two [Availability Zones (AZs)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). Public subnets will require a Route Table keep and an associated +two [Availability Zones (AZs)](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html). Public subnets require a Route Table keep and an associated Internet Gateway. ### Creating the Virtual Private Cloud (VPC) -We'll now create a VPC, a virtual networking environment that you'll control: +We now create a VPC, a virtual networking environment that you control: 1. Sign in to [Amazon Web Services](https://console.aws.amazon.com/vpc/home). 1. Select **Your VPCs** from the left menu and then select **Create VPC**. @@ -153,15 +153,15 @@ We'll now create a VPC, a virtual networking environment that you'll control: Now, let's create some subnets in different Availability Zones. Make sure that each subnet is associated to the VPC we just created and -that CIDR blocks don't overlap. This will also -allow us to enable multi AZ for redundancy. +that CIDR blocks don't overlap. This also +allows us to enable multi AZ for redundancy. -We will create private and public subnets to match load balancers and +We create private and public subnets to match load balancers and RDS instances as well: 1. Select **Subnets** from the left menu. 1. Select **Create subnet**. Give it a descriptive name tag based on the IP, - for example `gitlab-public-10.0.0.0`, select the VPC we created previously, select an availability zone (we'll use `us-west-2a`), + for example `gitlab-public-10.0.0.0`, select the VPC we created previously, select an availability zone (we use `us-west-2a`), and at the IPv4 CIDR block let's give it a 24 subnet `10.0.0.0/24`: ![Create subnet](img/create_subnet.png) @@ -186,7 +186,7 @@ create a new one: 1. Select **Internet Gateways** from the left menu. 1. Select **Create internet gateway**, give it the name `gitlab-gateway` and select **Create**. -1. Select it from the table, and then under the **Actions** dropdown choose +1. Select it from the table, and then under the **Actions** dropdown list choose "Attach to VPC". ![Create gateway](img/create_gateway.png) @@ -195,11 +195,11 @@ create a new one: ### Create NAT Gateways -Instances deployed in our private subnets need to connect to the internet for updates, but should not be reachable from the public internet. To achieve this, we'll make use of [NAT Gateways](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) deployed in each of our public subnets: +Instances deployed in our private subnets must connect to the internet for updates, but should not be reachable from the public internet. To achieve this, we make use of [NAT Gateways](https://docs.aws.amazon.com/vpc/latest/userguide/vpc-nat-gateway.html) deployed in each of our public subnets: -1. Navigate to the VPC dashboard and select **NAT Gateways** in the left menu bar. +1. Go to the VPC dashboard and select **NAT Gateways** in the left menu bar. 1. Select **Create NAT Gateway** and complete the following: - 1. **Subnet**: Select `gitlab-public-10.0.0.0` from the dropdown. + 1. **Subnet**: Select `gitlab-public-10.0.0.0` from the dropdown list. 1. **Elastic IP Allocation ID**: Enter an existing Elastic IP or select **Allocate Elastic IP address** to allocate a new IP to your NAT gateway. 1. Add tags if needed. 1. Select **Create NAT Gateway**. @@ -210,7 +210,7 @@ Create a second NAT gateway but this time place it in the second public subnet, #### Public Route Table -We need to create a route table for our public subnets to reach the internet via the internet gateway we created in the previous step. +We must create a route table for our public subnets to reach the internet via the internet gateway we created in the previous step. On the VPC dashboard: @@ -219,7 +219,7 @@ On the VPC dashboard: 1. At the "Name tag" enter `gitlab-public` and choose `gitlab-vpc` under "VPC". 1. Select **Create**. -We now need to add our internet gateway as a new target and have +We now must add our internet gateway as a new target and have it receive traffic from any destination. 1. Select **Route Tables** from the left menu and select the `gitlab-public` @@ -235,7 +235,7 @@ Next, we must associate the **public** subnets to the route table: #### Private Route Tables -We also need to create two private route tables so that instances in each private subnet can reach the internet via the NAT gateway in the corresponding public subnet in the same availability zone. +We also must create two private route tables so that instances in each private subnet can reach the internet via the NAT gateway in the corresponding public subnet in the same availability zone. 1. Follow the same steps as above to create two private route tables. Name them `gitlab-private-a` and `gitlab-private-b` respectively. 1. Next, add a new route to each of the private route tables where the destination is `0.0.0.0/0` and the target is one of the NAT gateways we created earlier. @@ -247,30 +247,30 @@ We also need to create two private route tables so that instances in each privat ## Load Balancer -We'll create a load balancer to evenly distribute inbound traffic on ports `80` and `443` across our GitLab application servers. Based the on the [scaling policies](#create-an-auto-scaling-group) we'll create later, instances will be added to or removed from our load balancer as needed. Additionally, the load balance will perform health checks on our instances. +We create a load balancer to evenly distribute inbound traffic on ports `80` and `443` across our GitLab application servers. Based on the [scaling policies](#create-an-auto-scaling-group) we create later, instances are added to or removed from our load balancer as needed. Additionally, the load balancer performs health checks on our instances. On the EC2 dashboard, look for Load Balancer in the left navigation bar: 1. Select **Create Load Balancer**. 1. Choose the **Classic Load Balancer**. - 1. Give it a name (we'll use `gitlab-loadbalancer`) and for the **Create LB Inside** option, select `gitlab-vpc` from the dropdown menu. + 1. Give it a name (we use `gitlab-loadbalancer`) and for the **Create LB Inside** option, select `gitlab-vpc` from the dropdown list. 1. In the **Listeners** section, set the following listeners: - HTTP port 80 for both load balancer and instance protocol and ports - TCP port 22 for both load balancer and instance protocols and ports - - HTTPS port 443 for load balancer protocol and ports, forwarding to HTTP port 80 on the instance (we will configure GitLab to listen on port 80 [later in the guide](#add-support-for-proxied-ssl)) + - HTTPS port 443 for load balancer protocol and ports, forwarding to HTTP port 80 on the instance (we configure GitLab to listen on port 80 [later in the guide](#add-support-for-proxied-ssl)) 1. In the **Select Subnets** section, select both public subnets from the list so that the load balancer can route traffic to both availability zones. -1. We'll add a security group for our load balancer to act as a firewall to control what traffic is allowed through. Select **Assign Security Groups** and select **Create a new security group**, give it a name - (we'll use `gitlab-loadbalancer-sec-group`) and description, and allow both HTTP and HTTPS traffic - from anywhere (`0.0.0.0/0, ::/0`). Also allow SSH traffic, select a custom source, and add a single trusted IP address or an IP address range in CIDR notation. This will allow users to perform Git actions over SSH. +1. We add a security group for our load balancer to act as a firewall to control what traffic is allowed through. Select **Assign Security Groups** and select **Create a new security group**, give it a name + (we use `gitlab-loadbalancer-sec-group`) and description, and allow both HTTP and HTTPS traffic + from anywhere (`0.0.0.0/0, ::/0`). Also allow SSH traffic, select a custom source, and add a single trusted IP address or an IP address range in CIDR notation. This allows users to perform Git actions over SSH. 1. Select **Configure Security Settings** and set the following: 1. Select an SSL/TLS certificate from ACM or upload a certificate to IAM. - 1. Under **Select a Cipher**, pick a predefined security policy from the dropdown. You can see a breakdown of [Predefined SSL Security Policies for Classic Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html) in the AWS docs. Check the GitLab codebase for a list of [supported SSL ciphers and protocols](https://gitlab.com/gitlab-org/gitlab/-/blob/9ee7ad433269b37251e0dd5b5e00a0f00d8126b4/lib/support/nginx/gitlab-ssl#L97-99). + 1. Under **Select a Cipher**, pick a predefined security policy from the dropdown list. You can see a breakdown of [Predefined SSL Security Policies for Classic Load Balancers](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-policy-table.html) in the AWS documentation. Check the GitLab codebase for a list of [supported SSL ciphers and protocols](https://gitlab.com/gitlab-org/gitlab/-/blob/9ee7ad433269b37251e0dd5b5e00a0f00d8126b4/lib/support/nginx/gitlab-ssl#L97-99). 1. Select **Configure Health Check** and set up a health check for your EC2 instances. 1. For **Ping Protocol**, select HTTP. 1. For **Ping Port**, enter 80. - 1. For **Ping Path** - we recommend that you [use the Readiness check endpoint](../../administration/load_balancer.md#readiness-check). You'll need to add [the VPC IP Address Range (CIDR)](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-groups.html#elb-vpc-nacl) to the [IP allowlist](../../administration/monitoring/ip_whitelist.md) for the [Health Check endpoints](../../user/admin_area/monitoring/health_check.md) + 1. For **Ping Path** - we recommend that you [use the Readiness check endpoint](../../administration/load_balancer.md#readiness-check). You must add [the VPC IP Address Range (CIDR)](https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-security-groups.html#elb-vpc-nacl) to the [IP allowlist](../../administration/monitoring/ip_whitelist.md) for the [Health Check endpoints](../../user/admin_area/monitoring/health_check.md) 1. Keep the default **Advanced Details** or adjust them according to your needs. -1. Select **Add EC2 Instances** - don't add anything as we will create an Auto Scaling Group later to manage instances for us. +1. Select **Add EC2 Instances** - don't add anything as we create an Auto Scaling Group later to manage instances for us. 1. Select **Add Tags** and add any tags you need. 1. Select **Review and Create**, review all your settings, and select **Create** if you're happy. @@ -288,28 +288,28 @@ On the Route 53 dashboard, select **Hosted zones** in the left navigation bar: 1. **Type:** Select **A - IPv4 address**. 1. **Alias:** Defaults to **No**. Select **Yes**. 1. **Alias Target:** Find the **ELB Classic Load Balancers** section and select the classic load balancer we created earlier. - 1. **Routing Policy:** We'll use **Simple** but you can choose a different policy based on your use case. - 1. **Evaluate Target Health:** We'll set this to **No** but you can choose to have the load balancer route traffic based on target health. + 1. **Routing Policy:** We use **Simple** but you can choose a different policy based on your use case. + 1. **Evaluate Target Health:** We set this to **No** but you can choose to have the load balancer route traffic based on target health. 1. Select **Create**. -1. If you registered your domain through Route 53, you're done. If you used a different domain registrar, you need to update your DNS records with your domain registrar. You'll need to: +1. If you registered your domain through Route 53, you're done. If you used a different domain registrar, you must update your DNS records with your domain registrar. You must: 1. Select **Hosted zones** and select the domain you added above. - 1. You'll see a list of `NS` records. From your domain registrar's administrator panel, add each of these as `NS` records to your domain's DNS records. These steps may vary between domain registrars. If you're stuck, Google **"name of your registrar" add DNS records** and you should find a help article specific to your domain registrar. + 1. You see a list of `NS` records. From your domain registrar's administrator panel, add each of these as `NS` records to your domain's DNS records. These steps may vary between domain registrars. If you're stuck, Google **"name of your registrar" add DNS records** and you should find a help article specific to your domain registrar. The steps for doing this vary depending on which registrar you use and is beyond the scope of this guide. ## PostgreSQL with RDS -For our database server we will use Amazon RDS for PostgreSQL which offers Multi AZ -for redundancy (Aurora is **not** supported). First we'll create a security group and subnet group, then we'll +For our database server we use Amazon RDS for PostgreSQL which offers Multi AZ +for redundancy (Aurora is **not** supported). First we create a security group and subnet group, then we create the actual RDS instance. ### RDS Security Group -We need a security group for our database that will allow inbound traffic from the instances we'll deploy in our `gitlab-loadbalancer-sec-group` later on: +We need a security group for our database that allows inbound traffic from the instances we deploy in our `gitlab-loadbalancer-sec-group` later on: 1. From the EC2 dashboard, select **Security Groups** from the left menu bar. 1. Select **Create security group**. -1. Give it a name (we'll use `gitlab-rds-sec-group`), a description, and select the `gitlab-vpc` from the **VPC** dropdown. +1. Give it a name (we use `gitlab-rds-sec-group`), a description, and select the `gitlab-vpc` from the **VPC** dropdown list. 1. In the **Inbound rules** section, select **Add rule** and set the following: 1. **Type:** search for and select the **PostgreSQL** rule. 1. **Source type:** set as "Custom". @@ -318,11 +318,11 @@ We need a security group for our database that will allow inbound traffic from t ### RDS Subnet Group -1. Navigate to the RDS dashboard and select **Subnet Groups** from the left menu. +1. Go to the RDS dashboard and select **Subnet Groups** from the left menu. 1. Select **Create DB Subnet Group**. -1. Under **Subnet group details**, enter a name (we'll use `gitlab-rds-group`), a description, and choose the `gitlab-vpc` from the VPC dropdown. -1. From the **Availability Zones** dropdown, select the Availability Zones that include the subnets you've configured. In our case, we'll add `eu-west-2a` and `eu-west-2b`. -1. From the **Subnets** dropdown, select the two private subnets (`10.0.1.0/24` and `10.0.3.0/24`) as we defined them in the [subnets section](#subnets). +1. Under **Subnet group details**, enter a name (we use `gitlab-rds-group`), a description, and choose the `gitlab-vpc` from the VPC dropdown list. +1. From the **Availability Zones** dropdown list, select the Availability Zones that include the subnets you've configured. In our case, we add `eu-west-2a` and `eu-west-2b`. +1. From the **Subnets** dropdown list, select the two private subnets (`10.0.1.0/24` and `10.0.3.0/24`) as we defined them in the [subnets section](#subnets). 1. Select **Create** when ready. ### Create the database @@ -332,30 +332,30 @@ Avoid using burstable instances (t class instances) for the database as this cou Now, it's time to create the database: -1. Navigate to the RDS dashboard, select **Databases** from the left menu, and select **Create database**. +1. Go to the RDS dashboard, select **Databases** from the left menu, and select **Create database**. 1. Select **Standard Create** for the database creation method. 1. Select **PostgreSQL** as the database engine and select the minimum PostgreSQL version as defined for your GitLab version in our [database requirements](../../install/requirements.md#postgresql-requirements). -1. Since this is a production server, let's choose **Production** from the **Templates** section. -1. Under **Settings**, set a DB instance identifier, a master username, and a master password. We'll use `gitlab-db-ha`, `gitlab`, and a very secure password respectively. Make a note of these as we'll need them later. -1. For the DB instance size, select **Standard classes** and select an instance size that meets your requirements from the dropdown menu. We'll use a `db.m4.large` instance. +1. Because this is a production server, let's choose **Production** from the **Templates** section. +1. Under **Settings**, set a DB instance identifier, a master username, and a master password. We use `gitlab-db-ha`, `gitlab`, and a very secure password respectively. Make a note of these as we need them later. +1. For the DB instance size, select **Standard classes** and select an instance size that meets your requirements from the dropdown list. We use a `db.m4.large` instance. 1. Under **Storage**, configure the following: - 1. Select **Provisioned IOPS (SSD)** from the storage type dropdown menu. Provisioned IOPS (SSD) storage is best suited for this use (though you can choose General Purpose (SSD) to reduce the costs). Read more about it at [Storage for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html). - 1. Allocate storage and set provisioned IOPS. We'll use the minimum values, `100` and `1000`, respectively. + 1. Select **Provisioned IOPS (SSD)** from the storage type dropdown list. Provisioned IOPS (SSD) storage is best suited for this use (though you can choose General Purpose (SSD) to reduce the costs). Read more about it at [Storage for Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Storage.html). + 1. Allocate storage and set provisioned IOPS. We use the minimum values, `100` and `1000`, respectively. 1. Enable storage autoscaling (optional) and set a maximum storage threshold. 1. Under **Availability & durability**, select **Create a standby instance** to have a standby RDS instance provisioned in a different [Availability Zone](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.MultiAZ.html). 1. Under **Connectivity**, configure the following: - 1. Select the VPC we created earlier (`gitlab-vpc`) from the **Virtual Private Cloud (VPC)** dropdown menu. + 1. Select the VPC we created earlier (`gitlab-vpc`) from the **Virtual Private Cloud (VPC)** dropdown list. 1. Expand the **Additional connectivity configuration** section and select the subnet group (`gitlab-rds-group`) we created earlier. 1. Set public accessibility to **No**. - 1. Under **VPC security group**, select **Choose existing** and select the `gitlab-rds-sec-group` we create above from the dropdown. + 1. Under **VPC security group**, select **Choose existing** and select the `gitlab-rds-sec-group` we create above from the dropdown list. 1. Leave the database port as the default `5432`. 1. For **Database authentication**, select **Password authentication**. 1. Expand the **Additional configuration** section and complete the following: - 1. The initial database name. We'll use `gitlabhq_production`. + 1. The initial database name. We use `gitlabhq_production`. 1. Configure your preferred backup settings. - 1. The only other change we'll make here is to disable auto minor version updates under **Maintenance**. + 1. The only other change we make here is to disable auto minor version updates under **Maintenance**. 1. Leave all the other settings as is or tweak according to your needs. - 1. Once you're happy, select **Create database**. + 1. If you're happy, select **Create database**. Now that the database is created, let's move on to setting up Redis with ElastiCache. @@ -366,17 +366,17 @@ persistence and is used to store session data, temporary cache information, and ### Create a Redis Security Group -1. Navigate to the EC2 dashboard. +1. Go to the EC2 dashboard. 1. Select **Security Groups** from the left menu. -1. Select **Create security group** and fill in the details. Give it a name (we'll use `gitlab-redis-sec-group`), +1. Select **Create security group** and fill in the details. Give it a name (we use `gitlab-redis-sec-group`), add a description, and choose the VPC we created previously 1. In the **Inbound rules** section, select **Add rule** and add a **Custom TCP** rule, set port `6379`, and set the "Custom" source as the `gitlab-loadbalancer-sec-group` we created earlier. 1. When done, select **Create security group**. ### Redis Subnet Group -1. Navigate to the ElastiCache dashboard from your AWS console. -1. Go to **Subnet Groups** in the left menu, and create a new subnet group (we'll name ours `gitlab-redis-group`). +1. Go to the ElastiCache dashboard from your AWS console. +1. Go to **Subnet Groups** in the left menu, and create a new subnet group (we name ours `gitlab-redis-group`). Make sure to select our VPC and its [private subnets](#subnets). 1. Select **Create** when ready. @@ -384,14 +384,14 @@ persistence and is used to store session data, temporary cache information, and ### Create the Redis Cluster -1. Navigate back to the ElastiCache dashboard. +1. Go back to the ElastiCache dashboard. 1. Select **Redis** on the left menu and select **Create** to create a new Redis cluster. Do not enable **Cluster Mode** as it is [not supported](../../administration/redis/replication_and_failover_external.md#requirements). Even without cluster mode on, you still get the chance to deploy Redis in multiple availability zones. 1. In the settings section: 1. Give the cluster a name (`gitlab-redis`) and a description. 1. For the version, select the latest. - 1. Leave the port as `6379` since this is what we used in our Redis security group above. + 1. Leave the port as `6379` because this is what we used in our Redis security group above. 1. Select the node type (at least `cache.t3.medium`, but adjust to your needs) and the number of replicas. 1. In the advanced settings section: 1. Select the multi-AZ auto-failover option. @@ -418,20 +418,20 @@ If you do not want to maintain bastion hosts, you can set up [AWS Systems Manage ### Create Bastion Host A -1. Navigate to the EC2 Dashboard and select **Launch instance**. +1. Go to the EC2 Dashboard and select **Launch instance**. 1. Select the **Ubuntu Server 18.04 LTS (HVM)** AMI. -1. Choose an instance type. We'll use a `t2.micro` as we'll only use the bastion host to SSH into our other instances. +1. Choose an instance type. We use a `t2.micro` as we only use the bastion host to SSH into our other instances. 1. Select **Configure Instance Details**. - 1. Under **Network**, select the `gitlab-vpc` from the dropdown menu. + 1. Under **Network**, select the `gitlab-vpc` from the dropdown list. 1. Under **Subnet**, select the public subnet we created earlier (`gitlab-public-10.0.0.0`). 1. Double check that under **Auto-assign Public IP** you have **Use subnet setting (Enable)** selected. 1. Leave everything else as default and select **Add Storage**. -1. For storage, we'll leave everything as default and only add an 8GB root volume. We won't store anything on this instance. +1. For storage, we leave everything as default and only add an 8GB root volume. We do not store anything on this instance. 1. Select **Add Tags** and on the next screen select **Add Tag**. - 1. We'll only set `Key: Name` and `Value: Bastion Host A`. + 1. We only set `Key: Name` and `Value: Bastion Host A`. 1. Select **Configure Security Group**. - 1. Select **Create a new security group**, enter a **Security group name** (we'll use `bastion-sec-group`), and add a description. - 1. We'll enable SSH access from anywhere (`0.0.0.0/0`). If you want stricter security, specify a single IP address or an IP address range in CIDR notation. + 1. Select **Create a new security group**, enter a **Security group name** (we use `bastion-sec-group`), and add a description. + 1. We enable SSH access from anywhere (`0.0.0.0/0`). If you want stricter security, specify a single IP address or an IP address range in CIDR notation. 1. Select **Review and Launch** 1. Review all your settings and, if you're happy, select **Launch**. 1. Acknowledge that you have access to an existing key pair or create a new one. Select **Launch Instance**. @@ -447,18 +447,18 @@ Confirm that you can SSH into the instance: 1. Create an EC2 instance following the same steps as above with the following changes: 1. For the **Subnet**, select the second public subnet we created earlier (`gitlab-public-10.0.2.0`). - 1. Under the **Add Tags** section, we'll set `Key: Name` and `Value: Bastion Host B` so that we can easily identify our two instances. + 1. Under the **Add Tags** section, we set `Key: Name` and `Value: Bastion Host B` so that we can easily identify our two instances. 1. For the security group, select the existing `bastion-sec-group` we created above. ### Use SSH Agent Forwarding -EC2 instances running Linux use private key files for SSH authentication. You'll connect to your bastion host using an SSH client and the private key file stored on your client. Since the private key file is not present on the bastion host, you will not be able to connect to your instances in private subnets. +EC2 instances running Linux use private key files for SSH authentication. You connect to your bastion host using an SSH client and the private key file stored on your client. Because the private key file is not present on the bastion host, you are not able to connect to your instances in private subnets. Storing private key files on your bastion host is a bad idea. To get around this, use SSH agent forwarding on your client. See [Securely Connect to Linux Instances Running in a Private Amazon VPC](https://aws.amazon.com/blogs/security/securely-connect-to-linux-instances-running-in-a-private-amazon-vpc/) for a step-by-step guide on how to use SSH agent forwarding. ## Install GitLab and create custom AMI -We will need a preconfigured, custom GitLab AMI to use in our launch configuration later. As a starting point, we will use the official GitLab AMI to create a GitLab instance. Then, we'll add our custom configuration for PostgreSQL, Redis, and Gitaly. If you prefer, instead of using the official GitLab AMI, you can also spin up an EC2 instance of your choosing and [manually install GitLab](https://about.gitlab.com/install/). +We need a preconfigured, custom GitLab AMI to use in our launch configuration later. As a starting point, we use the official GitLab AMI to create a GitLab instance. Then, we add our custom configuration for PostgreSQL, Redis, and Gitaly. If you prefer, instead of using the official GitLab AMI, you can also spin up an EC2 instance of your choosing and [manually install GitLab](https://about.gitlab.com/install/). ### Install GitLab @@ -467,12 +467,12 @@ From the EC2 dashboard: 1. Use the section below titled "[Find official GitLab-created AMI IDs on AWS](#find-official-gitlab-created-ami-ids-on-aws)" to find the correct AMI to launch. 1. After clicking **Launch** on the desired AMI, select an instance type based on your workload. Consult the [hardware requirements](../../install/requirements.md#hardware-requirements) to choose one that fits your needs (at least `c5.xlarge`, which is sufficient to accommodate 100 users). 1. Select **Configure Instance Details**: - 1. In the **Network** dropdown, select `gitlab-vpc`, the VPC we created earlier. - 1. In the **Subnet** dropdown, select `gitlab-private-10.0.1.0` from the list of subnets we created earlier. + 1. In the **Network** dropdown list, select `gitlab-vpc`, the VPC we created earlier. + 1. In the **Subnet** dropdown list, select `gitlab-private-10.0.1.0` from the list of subnets we created earlier. 1. Double check that **Auto-assign Public IP** is set to `Use subnet setting (Disable)`. 1. Select **Add Storage**. - 1. The root volume is 8GiB by default and should be enough given that we won't store any data there. -1. Select **Add Tags** and add any tags you may need. In our case, we'll only set `Key: Name` and `Value: GitLab`. + 1. The root volume is 8GiB by default and should be enough given that we do not store any data there. +1. Select **Add Tags** and add any tags you may need. In our case, we only set `Key: Name` and `Value: GitLab`. 1. Select **Configure Security Group**. Check **Select an existing security group** and select the `gitlab-loadbalancer-sec-group` we created earlier. 1. Select **Review and launch** followed by **Launch** if you're happy with your settings. 1. Finally, acknowledge that you have access to the selected private key file or create a new one. Select **Launch Instances**. @@ -483,7 +483,7 @@ Connect to your GitLab instance via **Bastion Host A** using [SSH Agent Forwardi #### Disable Let's Encrypt -Since we're adding our SSL certificate at the load balancer, we do not need the GitLab built-in support for Let's Encrypt. Let's Encrypt [is enabled by default](https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration) when using an `https` domain in GitLab 10.7 and later, so we need to explicitly disable it: +Because we're adding our SSL certificate at the load balancer, we do not need the GitLab built-in support for Let's Encrypt. Let's Encrypt [is enabled by default](https://docs.gitlab.com/omnibus/settings/ssl.html#lets-encrypt-integration) when using an `https` domain in GitLab 10.7 and later, so we must explicitly disable it: 1. Open `/etc/gitlab/gitlab.rb` and disable it: @@ -501,7 +501,7 @@ Since we're adding our SSL certificate at the load balancer, we do not need the From your GitLab instance, connect to the RDS instance to verify access and to install the required `pg_trgm` and `btree_gist` extensions. -To find the host or endpoint, navigate to **Amazon RDS > Databases** and select the database you created earlier. Look for the endpoint under the **Connectivity & security** tab. +To find the host or endpoint, go to **Amazon RDS > Databases** and select the database you created earlier. Look for the endpoint under the **Connectivity & security** tab. Do not to include the colon and port number: @@ -523,10 +523,10 @@ gitlab=# \q #### Configure GitLab to connect to PostgreSQL and Redis 1. Edit `/etc/gitlab/gitlab.rb`, find the `external_url 'http://<domain>'` option - and change it to the `https` domain you will be using. + and change it to the `https` domain you are using. 1. Look for the GitLab database settings and uncomment as necessary. In - our current case we'll specify the database adapter, encoding, host, name, + our current case we specify the database adapter, encoding, host, name, username, and password: ```ruby @@ -542,7 +542,7 @@ gitlab=# \q gitlab_rails['db_host'] = "<rds-endpoint>" ``` -1. Next, we need to configure the Redis section by adding the host and +1. Next, we must configure the Redis section by adding the host and uncommenting the port: ```ruby @@ -560,7 +560,7 @@ gitlab=# \q sudo gitlab-ctl reconfigure ``` -1. You might also find it useful to run a check and a service status to make sure +1. You can also run a check and a service status to make sure everything has been setup correctly: ```shell @@ -578,21 +578,21 @@ Gitaly is a service that provides high-level RPC access to Git repositories. It should be enabled and configured on a separate EC2 instance in one of the [private subnets](#subnets) we configured previously. -Let's create an EC2 instance where we'll install Gitaly: +Let's create an EC2 instance where we install Gitaly: 1. From the EC2 dashboard, select **Launch instance**. -1. Choose an AMI. In this example, we'll select the **Ubuntu Server 18.04 LTS (HVM), SSD Volume Type**. -1. Choose an instance type. We'll pick a `c5.xlarge`. +1. Choose an AMI. In this example, we select the **Ubuntu Server 18.04 LTS (HVM), SSD Volume Type**. +1. Choose an instance type. We pick a `c5.xlarge`. 1. Select **Configure Instance Details**. - 1. In the **Network** dropdown, select `gitlab-vpc`, the VPC we created earlier. - 1. In the **Subnet** dropdown, select `gitlab-private-10.0.1.0` from the list of subnets we created earlier. + 1. In the **Network** dropdown list, select `gitlab-vpc`, the VPC we created earlier. + 1. In the **Subnet** dropdown list, select `gitlab-private-10.0.1.0` from the list of subnets we created earlier. 1. Double check that **Auto-assign Public IP** is set to `Use subnet setting (Disable)`. 1. Select **Add Storage**. 1. Increase the Root volume size to `20 GiB` and change the **Volume Type** to `Provisioned IOPS SSD (io1)`. (This is an arbitrary size. Create a volume big enough for your repository storage requirements.) 1. For **IOPS** set `1000` (20 GiB x 50 IOPS). You can provision up to 50 IOPS per GiB. If you select a larger volume, increase the IOPS accordingly. Workloads where many small files are written in a serialized manner, like `git`, requires performant storage, hence the choice of `Provisioned IOPS SSD (io1)`. -1. Select **Add Tags** and add your tags. In our case, we'll only set `Key: Name` and `Value: Gitaly`. +1. Select **Add Tags** and add your tags. In our case, we only set `Key: Name` and `Value: Gitaly`. 1. Select **Configure Security Group** and let's **Create a new security group**. - 1. Give your security group a name and description. We'll use `gitlab-gitaly-sec-group` for both. + 1. Give your security group a name and description. We use `gitlab-gitaly-sec-group` for both. 1. Create a **Custom TCP** rule and add port `8075` to the **Port Range**. For the **Source**, select the `gitlab-loadbalancer-sec-group`. 1. Also add an inbound rule for SSH from the `bastion-sec-group` so that we can connect using [SSH Agent Forwarding](#use-ssh-agent-forwarding) from the Bastion hosts. 1. Select **Review and launch** followed by **Launch** if you're happy with your settings. @@ -611,11 +611,11 @@ Remember to run `sudo gitlab-ctl reconfigure` after saving the changes to the `g #### Fast lookup of authorized SSH keys -The public SSH keys for users allowed to access GitLab are stored in `/var/opt/gitlab/.ssh/authorized_keys`. Typically we'd use shared storage so that all the instances are able to access this file when a user performs a Git action over SSH. Since we do not have shared storage in our setup, we'll update our configuration to authorize SSH users via indexed lookup in the GitLab database. +The public SSH keys for users allowed to access GitLab are stored in `/var/opt/gitlab/.ssh/authorized_keys`. Typically we'd use shared storage so that all the instances are able to access this file when a user performs a Git action over SSH. Because we do not have shared storage in our setup, we update our configuration to authorize SSH users via indexed lookup in the GitLab database. Follow the instructions at [Setting up fast lookup via GitLab Shell](../../administration/operations/fast_ssh_key_lookup.md#setting-up-fast-lookup-via-gitlab-shell) to switch from using the `authorized_keys` file to the database. -If you do not configure fast lookup, Git actions over SSH will result in the following error: +If you do not configure fast lookup, Git actions over SSH results in the following error: ```shell Permission denied (publickey). @@ -629,7 +629,7 @@ and the repository exists. Ordinarily we would manually copy the contents (primary and public keys) of `/etc/ssh/` on the primary application server to `/etc/ssh` on all secondary servers. This prevents false man-in-the-middle-attack alerts when accessing servers in your cluster behind a load balancer. -We'll automate this by creating static host keys as part of our custom AMI. As these host keys are also rotated every time an EC2 instance boots up, "hard coding" them into our custom AMI serves as a handy workaround. +We automate this by creating static host keys as part of our custom AMI. As these host keys are also rotated every time an EC2 instance boots up, "hard coding" them into our custom AMI serves as a workaround. On your GitLab instance run the following: @@ -650,16 +650,16 @@ HostKey /etc/ssh_static/ssh_host_ed25519_key #### Amazon S3 object storage -Since we're not using NFS for shared storage, we will use [Amazon S3](https://aws.amazon.com/s3/) buckets to store backups, artifacts, LFS objects, uploads, merge request diffs, container registry images, and more. Our documentation includes [instructions on how to configure object storage](../../administration/object_storage.md) for each of these data types, and other information about using object storage with GitLab. +Because we're not using NFS for shared storage, we use [Amazon S3](https://aws.amazon.com/s3/) buckets to store backups, artifacts, LFS objects, uploads, merge request diffs, container registry images, and more. Our documentation includes [instructions on how to configure object storage](../../administration/object_storage.md) for each of these data types, and other information about using object storage with GitLab. NOTE: -Since we are using the [AWS IAM profile](#create-an-iam-role) we created earlier, be sure to omit the AWS access key and secret access key/value pairs when configuring object storage. Instead, use `'use_iam_profile' => true` in your configuration as shown in the object storage documentation linked above. +Because we are using the [AWS IAM profile](#create-an-iam-role) we created earlier, be sure to omit the AWS access key and secret access key/value pairs when configuring object storage. Instead, use `'use_iam_profile' => true` in your configuration as shown in the object storage documentation linked above. Remember to run `sudo gitlab-ctl reconfigure` after saving the changes to the `gitlab.rb` file. --- -That concludes the configuration changes for our GitLab instance. Next, we'll create a custom AMI based on this instance to use for our launch configuration and auto scaling group. +That concludes the configuration changes for our GitLab instance. Next, we create a custom AMI based on this instance to use for our launch configuration and auto scaling group. ### Log in for the first time @@ -672,7 +672,7 @@ Depending on how you installed GitLab and if you did not change the password by To change the default password, log in as the `root` user with the default password and [change it in the user profile](../../user/profile#change-your-password). -When our [auto scaling group](#create-an-auto-scaling-group) spins up new instances, we'll be able to log in with username `root` and the newly created password. +When our [auto scaling group](#create-an-auto-scaling-group) spins up new instances, we are able to log in with username `root` and the newly created password. ### Create custom AMI @@ -680,10 +680,10 @@ On the EC2 dashboard: 1. Select the `GitLab` instance we [created earlier](#install-gitlab). 1. Select **Actions**, scroll down to **Image** and select **Create Image**. -1. Give your image a name and description (we'll use `GitLab-Source` for both). +1. Give your image a name and description (we use `GitLab-Source` for both). 1. Leave everything else as default and select **Create Image** -Now we have a custom AMI that we'll use to create our launch configuration the next step. +Now we have a custom AMI that we use to create our launch configuration the next step. ## Deploy GitLab inside an auto scaling group @@ -694,11 +694,11 @@ From the EC2 dashboard: 1. Select **Launch Configurations** from the left menu and select **Create launch configuration**. 1. Select **My AMIs** from the left menu and select the `GitLab` custom AMI we created above. 1. Select an instance type best suited for your needs (at least a `c5.xlarge`) and select **Configure details**. -1. Enter a name for your launch configuration (we'll use `gitlab-ha-launch-config`). +1. Enter a name for your launch configuration (we use `gitlab-ha-launch-config`). 1. **Do not** check **Request Spot Instance**. -1. From the **IAM Role** dropdown, pick the `GitLabAdmin` instance role we [created earlier](#create-an-iam-ec2-instance-role-and-profile). +1. From the **IAM Role** dropdown list, pick the `GitLabAdmin` instance role we [created earlier](#create-an-iam-ec2-instance-role-and-profile). 1. Leave the rest as defaults and select **Add Storage**. -1. The root volume is 8GiB by default and should be enough given that we won't store any data there. Select **Configure Security Group**. +1. The root volume is 8GiB by default and should be enough given that we do not store any data there. Select **Configure Security Group**. 1. Check **Select and existing security group** and select the `gitlab-loadbalancer-sec-group` we created earlier. 1. Select **Review**, review your changes, and select **Create launch configuration**. 1. Acknowledge that you have access to the private key or create a new one. Select **Create launch configuration**. @@ -706,16 +706,16 @@ From the EC2 dashboard: ### Create an auto scaling group 1. After the launch configuration is created, select **Create an Auto Scaling group using this launch configuration** to start creating the auto scaling group. -1. Enter a **Group name** (we'll use `gitlab-auto-scaling-group`). -1. For **Group size**, enter the number of instances you want to start with (we'll enter `2`). -1. Select the `gitlab-vpc` from the **Network** dropdown. +1. Enter a **Group name** (we use `gitlab-auto-scaling-group`). +1. For **Group size**, enter the number of instances you want to start with (we enter `2`). +1. Select the `gitlab-vpc` from the **Network** dropdown list. 1. Add both the private [subnets we created earlier](#subnets). 1. Expand the **Advanced Details** section and check the **Receive traffic from one or more load balancers** option. -1. From the **Classic Load Balancers** dropdown, select the load balancer we created earlier. +1. From the **Classic Load Balancers** dropdown list, select the load balancer we created earlier. 1. For **Health Check Type**, select **ELB**. -1. We'll leave our **Health Check Grace Period** as the default `300` seconds. Select **Configure scaling policies**. +1. We leave our **Health Check Grace Period** as the default `300` seconds. Select **Configure scaling policies**. 1. Check **Use scaling policies to adjust the capacity of this group**. -1. For this group we'll scale between 2 and 4 instances where one instance will be added if CPU +1. For this group we scale between 2 and 4 instances where one instance is added if CPU utilization is greater than 60% and one instance is removed if it falls to less than 45%. @@ -724,9 +724,9 @@ to less than 45%. 1. Finally, configure notifications and tags as you see fit, review your changes, and create the auto scaling group. -As the auto scaling group is created, you'll see your new instances spinning up in your EC2 dashboard. You'll also see the new instances added to your load balancer. Once the instances pass the heath check, they are ready to start receiving traffic from the load balancer. +As the auto scaling group is created, you see your new instances spinning up in your EC2 dashboard. You also see the new instances added to your load balancer. After the instances pass the heath check, they are ready to start receiving traffic from the load balancer. -Since our instances are created by the auto scaling group, go back to your instances and terminate the [instance we created manually above](#install-gitlab). We only needed this instance to create our custom AMI. +Because our instances are created by the auto scaling group, go back to your instances and terminate the [instance we created manually above](#install-gitlab). We only needed this instance to create our custom AMI. ## Health check and monitoring with Prometheus @@ -753,11 +753,11 @@ and restore its Git data, database, attachments, LFS objects, and so on. Some important things to know: -- The backup/restore tool **does not** store some configuration files, like secrets; you'll - need to [configure this yourself](../../raketasks/backup_restore.md#storing-configuration-files). +- The backup/restore tool **does not** store some configuration files, like secrets; you + must [configure this yourself](../../raketasks/backup_gitlab.md#storing-configuration-files). - By default, the backup files are stored locally, but you can - [backup GitLab using S3](../../raketasks/backup_restore.md#using-amazon-s3). -- You can [exclude specific directories form the backup](../../raketasks/backup_restore.md#excluding-specific-directories-from-the-backup). + [backup GitLab using S3](../../raketasks/backup_gitlab.md#using-amazon-s3). +- You can [exclude specific directories form the backup](../../raketasks/backup_gitlab.md#excluding-specific-directories-from-the-backup). ### Backing up GitLab @@ -777,7 +777,7 @@ For GitLab 12.1 and earlier, use `gitlab-rake gitlab:backup:create`. To restore GitLab, first review the [restore documentation](../../raketasks/backup_restore.md#restore-gitlab), and primarily the restore prerequisites. Then, follow the steps under the -[Omnibus installations section](../../raketasks/backup_restore.md#restore-for-omnibus-gitlab-installations). +[Omnibus installations section](../../raketasks/restore_gitlab.md#restore-for-omnibus-gitlab-installations). ## Updating GitLab @@ -825,7 +825,7 @@ to request additional material: GitLab supports several different types of clustering. - [Geo replication](../../administration/geo/index.md): Geo is the solution for widely distributed development teams. -- [Omnibus GitLab](https://docs.gitlab.com/omnibus/) - Everything you need to know +- [Omnibus GitLab](https://docs.gitlab.com/omnibus/) - Everything you must know about administering your GitLab instance. - [Add a license](../../user/admin_area/license.md): Activate all GitLab Enterprise Edition functionality with a license. @@ -835,9 +835,9 @@ to request additional material: ### Instances are failing health checks -If your instances are failing the load balancer's health checks, verify that they are returning a status `200` from the health check endpoint we configured earlier. Any other status, including redirects like status `302`, will cause the health check to fail. +If your instances are failing the load balancer's health checks, verify that they are returning a status `200` from the health check endpoint we configured earlier. Any other status, including redirects like status `302`, causes the health check to fail. -You may have to set a password on the `root` user to prevent automatic redirects on the sign-in endpoint before health checks will pass. +You may have to set a password on the `root` user to prevent automatic redirects on the sign-in endpoint before health checks pass. ### "The change you requested was rejected (422)" diff --git a/doc/install/docker.md b/doc/install/docker.md index 058233520ca..356c025e168 100644 --- a/doc/install/docker.md +++ b/doc/install/docker.md @@ -272,6 +272,10 @@ Here's an example that deploys GitLab with four runners as a [stack](https://doc docker stack deploy --compose-file docker-compose.yml mystack ``` +### Install the product documentation + +This is an optional step. See how to [self-host the product documentation](../administration/docs_self_host.md#self-host-the-product-documentation-with-docker). + ## Configuration This container uses the official Omnibus GitLab package, so all configuration @@ -529,6 +533,11 @@ The following steps assume that you are upgrading the same version. replace `ce` with `ee` in your `docker run` command or `docker-compose.yml` file. However, reuse the CE container name, port and file mappings, and version. +### Upgrade the product documentation + +This is an optional step. If you [installed the documentation site](#install-the-product-documentation), +see how to [upgrade to another version](../administration/docs_self_host.md#upgrade-using-docker). + ## Back up GitLab You can create a GitLab backup with: diff --git a/doc/install/installation.md b/doc/install/installation.md index cc2e57aac96..2f2ae016edd 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -27,20 +27,20 @@ following the ## Consider the Omnibus package installation -Since an installation from source is a lot of work and error prone we strongly recommend the fast and reliable [Omnibus package installation](https://about.gitlab.com/install/) (deb/rpm). +Because an installation from source is a lot of work and error prone we strongly recommend the fast and reliable [Omnibus package installation](https://about.gitlab.com/install/) (deb/rpm). One reason the Omnibus package is more reliable is its use of runit to restart any of the GitLab processes in case one crashes. On heavily used GitLab instances the memory usage of the Sidekiq background worker grows over time. Omnibus packages solve this by [letting the Sidekiq terminate gracefully](../administration/operations/sidekiq_memory_killer.md) if it uses too much memory. After this termination runit detects Sidekiq is not running and starts it. -Since installations from source don't use runit for process supervision, Sidekiq +Because installations from source don't use runit for process supervision, Sidekiq can't be terminated and its memory usage grows over time. ## Select a version to install Make sure you view [this installation guide](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/install/installation.md) from the branch (version) of GitLab you would like to install (for example, `11-7-stable`). -You can select the branch in the version dropdown in the top left corner of GitLab (below the menu bar). +You can select the branch in the version dropdown list in the top left corner of GitLab (below the menu bar). If the highest number stable branch is unclear, check the [GitLab blog](https://about.gitlab.com/blog/) for installation guide links by version. @@ -51,7 +51,7 @@ If the highest number stable branch is unclear, check the [GitLab blog](https:// | [Ruby](#2-ruby) | `2.7` | From GitLab 13.6, Ruby 2.7 is required. Ruby 3.0 is not supported yet (see [the relevant epic](https://gitlab.com/groups/gitlab-org/-/epics/5149) for the current status). You must use the standard MRI implementation of Ruby. We love [JRuby](https://www.jruby.org/) and [Rubinius](https://github.com/rubinius/rubinius#the-rubinius-language-platform), but GitLab needs several Gems that have native extensions. | | [Go](#3-go) | `1.16` | | | [Git](#git) | `2.33.x` | From GitLab 14.4, Git 2.33.x and later is required. It's highly recommended that you use the [Git version provided by Gitaly](#git). | -| [Node.js](#4-node) | `14.15.0` | GitLab uses [webpack](https://webpack.js.org/) to compile frontend assets. Node.js 16.x is recommended, as it's faster. You can check which version you're running with `node -v`. You need to update it to a newer version if needed. | +| [Node.js](#4-node) | `14.15.0` | GitLab uses [webpack](https://webpack.js.org/) to compile frontend assets. Node.js 16.x is recommended, as it's faster. You can check which version you're running with `node -v`. You must update it to a newer version if needed. | ## GitLab directory structure @@ -179,7 +179,7 @@ the Git path: ### GraphicsMagick For the [Custom Favicon](../user/admin_area/appearance.md#favicon) to work, GraphicsMagick -needs to be installed. +must be installed. ```shell sudo apt-get install -y graphicsmagick @@ -187,7 +187,7 @@ sudo apt-get install -y graphicsmagick ### Mail server -In order to receive mail notifications, make sure to install a mail server. +To receive mail notifications, make sure to install a mail server. By default, Debian is shipped with `exim4` but this [has problems](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/12754) while Ubuntu does not ship with one. The recommended mail server is `postfix` and you @@ -264,7 +264,7 @@ requirements for these are: In many distributions, the versions provided by the official package repositories are out of date, so -we need to install through the following commands: +we must install through the following commands: ```shell # install node v16.x @@ -298,7 +298,7 @@ In GitLab 12.1 and later, only PostgreSQL is supported. In GitLab 14.0 and later ``` For Ubuntu 18.04 and earlier, the available PostgreSQL doesn't meet the minimum - version requirement. You need to add PostgreSQL's repository: + version requirement. You must add PostgreSQL's repository: ```shell wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - @@ -426,7 +426,7 @@ sudo usermod -aG redis git ### Supervise Redis with systemd If your distribution uses systemd init and the output of the following command is `notify`, -you do not need to make any changes: +you must not make any changes: ```shell systemctl show --value --property=Type redis-server.service @@ -678,7 +678,7 @@ sudo -u git -H bundle exec rake "gitlab:indexer:install[/home/git/gitlab-elastic ``` The source code first is fetched to the path specified by the first parameter. Then a binary is built under its `bin` directory. -You then need to update `gitlab.yml`'s `production -> elasticsearch -> indexer_path` setting to point to that binary. +You must then update `gitlab.yml`'s `production -> elasticsearch -> indexer_path` setting to point to that binary. ### Install GitLab Pages @@ -740,7 +740,7 @@ sudo systemctl daemon-reload The units provided by GitLab make very little assumptions about where you are running Redis and PostgreSQL. -If you installed GitLab in another directory or as a user other than the default, you need to change these values in the units as well. +If you installed GitLab in another directory or as a user other than the default, you must change these values in the units as well. For example, if you're running Redis and PostgreSQL on the same machine as GitLab, you should: @@ -853,7 +853,7 @@ sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production force=yes # When done, you see 'Administrator account created:' ``` -You can set the Administrator/root password and email by supplying them in environmental variables, `GITLAB_ROOT_PASSWORD` and `GITLAB_ROOT_EMAIL` respectively, as seen below. If you don't set the password (and it is set to the default one), wait to expose GitLab to the public internet until the installation is done and you've logged into the server the first time. During the first login, you'll be forced to change the default password. An Enterprise Edition license may also be installed at this time by supplying a full path in the `GITLAB_LICENSE_FILE` environment variable. +You can set the Administrator/root password and email by supplying them in environmental variables, `GITLAB_ROOT_PASSWORD` and `GITLAB_ROOT_EMAIL` respectively, as seen below. If you don't set the password (and it is set to the default one), wait to expose GitLab to the public internet until the installation is done and you've logged into the server the first time. During the first login, you are forced to change the default password. An Enterprise Edition license may also be installed at this time by supplying a full path in the `GITLAB_LICENSE_FILE` environment variable. ```shell sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=yourpassword GITLAB_ROOT_EMAIL=youremail GITLAB_LICENSE_FILE="/path/to/license" @@ -942,7 +942,7 @@ to use. Read all about the needed configuration at the If you want to use HTTPS, replace the `gitlab` NGINX configuration with `gitlab-ssl`. See [Using HTTPS](#using-https) for HTTPS configuration details. -For the NGINX to be able to read the GitLab-Workhorse socket, you need to make sure, that the `www-data` user can read the socket, which will be owned by the GitLab user. This is most easily achieved, if it is world-readable, for example that it has permissions `0755`, which is the default. `www-data` also needs to be able to list the parent directories. +For the NGINX to be able to read the GitLab-Workhorse socket, you must make sure, that the `www-data` user can read the socket, which is owned by the GitLab user. This is achieved, if it is world-readable, for example that it has permissions `0755`, which is the default. `www-data` also must be able to list the parent directories. ### Test Configuration @@ -999,8 +999,8 @@ Supply the `SANITIZE=true` environment variable to `gitlab:check` to omit projec Visit YOUR_SERVER in your web browser for your first GitLab login. If you didn't [provide a root password during setup](#initialize-database-and-activate-advanced-features), -you'll be redirected to a password reset screen to provide the password for the -initial administrator account. Enter your desired password and you'll be +you are redirected to a password reset screen to provide the password for the +initial administrator account. Enter your desired password and you are redirected back to the login screen. The default account's username is **root**. Provide the password you created @@ -1013,6 +1013,10 @@ To start and stop GitLab when using: - systemd units: use `sudo systemctl start gitlab.target` or `sudo systemctl stop gitlab.target`. - The SysV init script: use `sudo service gitlab start` or `sudo service gitlab stop`. +### Install the product documentation + +This is an optional step. See how to [self-host the product documentation](../administration/docs_self_host.md). + ## Advanced Setup Tips ### Relative URL support @@ -1110,7 +1114,7 @@ host localhost # Give your setup a name (here: override localhost) hostname 127.0.0.1; # Your server name or IP ``` -You also need to change the corresponding options (for example, `ssh_user`, `ssh_host`, `admin_uri`) in the `config/gitlab.yml` file. +You must also change the corresponding options (for example, `ssh_user`, `ssh_host`, `admin_uri`) in the `config/gitlab.yml` file. ### Additional Markup Styles @@ -1119,7 +1123,7 @@ Apart from the always supported Markdown style, there are other rich text files ### Using Sidekiq instead of Sidekiq Cluster As of GitLab 12.10, Source installations are using `bin/sidekiq-cluster` for managing Sidekiq processes. -Using Sidekiq directly is still supported until 14.0. So if you're experiencing issues, please: +Using Sidekiq directly is still supported until 14.0. So if you're experiencing issues: 1. Edit the system `init.d` script to remove the `SIDEKIQ_WORKERS` flag. If you have `/etc/default/gitlab`, then you should edit it instead. 1. Restart GitLab. diff --git a/doc/install/requirements.md b/doc/install/requirements.md index d01860cb24f..27148e06ccb 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -26,22 +26,22 @@ For the installation options, see [the main installation page](index.md). - macOS Installation of GitLab on these operating systems is possible, but not supported. -Please see the [installation from source guide](installation.md) and the [installation guides](https://about.gitlab.com/install/) for more information. +See the [installation from source guide](installation.md) and the [installation guides](https://about.gitlab.com/install/) for more information. -Please see [OS versions that are no longer supported](../administration/package_information/supported_os.md#os-versions-that-are-no-longer-supported) for Omnibus installs page +See [OS versions that are no longer supported](../administration/package_information/supported_os.md#os-versions-that-are-no-longer-supported) for Omnibus installs page for a list of supported and unsupported OS versions as well as the last support GitLab version for that OS. ### Microsoft Windows GitLab is developed for Linux-based operating systems. It does **not** run on Microsoft Windows, and we have no plans to support it in the near future. For the latest development status view this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/22337). -Please consider using a virtual machine to run GitLab. +Consider using a virtual machine to run GitLab. ## Software requirements ### Redis versions -GitLab 13.0 and later requires Redis version 4.0 or higher. +GitLab 13.0 and later requires Redis version 5.0 or higher. Redis version 6.0 or higher is recommended, as this is what ships with [Omnibus GitLab](https://docs.gitlab.com/omnibus/) packages starting with GitLab 13.9. @@ -61,7 +61,7 @@ Apart from a local hard drive you can also mount a volume that supports the netw If you have enough RAM and a recent CPU the speed of GitLab is mainly limited by hard drive seek times. Having a fast drive (7200 RPM and up) or a solid state drive (SSD) improves the responsiveness of GitLab. NOTE: -Since file system performance may affect the overall performance of GitLab, +Because file system performance may affect the overall performance of GitLab, [we don't recommend using cloud-based file systems for storage](../administration/nfs.md#avoid-using-cloud-based-file-systems). NOTE: @@ -94,6 +94,14 @@ if your available memory changes. We also recommend configuring the kernel's swa to a low value like `10` to make the most of your RAM while still having the swap available when needed. +NOTE: +Although excessive swapping is undesired and degrades performance, it is an +extremely important last resort against out-of-memory conditions. During +unexpected system load, such as OS updates or other services on the same host, +peak memory load spikes could be much higher than average. Having plenty of swap +helps avoid the Linux OOM killer unsafely terminating a potentially critical +process, such as PostgreSQL, which can have disastrous consequences. + ## Database PostgreSQL is the only supported database, which is bundled with the Omnibus GitLab package. @@ -112,7 +120,7 @@ the following table) as these were used for development and testing: | GitLab version | Minimum PostgreSQL version | |----------------|----------------------------| | 13.0 | 11 | -| 14.0 | 12.10 | +| 14.0 | 12.7 | | 15.0 | 12.10 | | 16.0 (planned) | 13.6 | @@ -125,6 +133,12 @@ GitLab database. [Read more about this requirement, and troubleshooting](postgre | `btree_gist` | 13.1 | | `plpgsql` | 11.7 | +The following managed PostgreSQL services are known to be incompatible and should not be used: + +| GitLab version | Managed service | +|----------------|-------------------------------------------------------| +| 14.4+ | Amazon Aurora (see [14.4.0](../update/index.md#1440)) | + NOTE: Support for [PostgreSQL 9.6 and 10 was removed in GitLab 13.0](https://about.gitlab.com/releases/2020/05/22/gitlab-13-0-released/#postgresql-11-is-now-the-minimum-required-version-to-install-gitlab) so that GitLab can benefit from PostgreSQL 11 improvements, such as partitioning. @@ -251,9 +265,9 @@ works. ### Puma per worker maximum memory -By default, each Puma worker will be limited to 1024 MB of memory. +By default, each Puma worker is limited to 1024 MB of memory. This setting [can be adjusted](../administration/operations/puma.md#change-the-memory-limit-setting) and should be considered -if you need to increase the number of Puma workers. +if you must increase the number of Puma workers. ## Redis and Sidekiq @@ -279,7 +293,7 @@ to install GitLab on. Depending on how you decide to configure GitLab Runner and what tools you use to exercise your application in the CI environment, GitLab Runner can consume significant amount of available memory. -Memory consumption calculations, that are available above, won't be valid if +Memory consumption calculations, that are available above, are not valid if you decide to run GitLab Runner and the GitLab Rails application on the same machine. @@ -295,7 +309,7 @@ The GitLab Runner server requirements depend on: - Resources required to run build jobs. - Job concurrency settings. -Since the nature of the jobs varies for each use case, you need to experiment by adjusting the job concurrency to get the optimum setting. +Because the nature of the jobs varies for each use case, you must experiment by adjusting the job concurrency to get the optimum setting. For reference, the [SaaS runners on Linux](../ci/runners/saas/linux_saas_runner.md) are configured so that a **single job** runs in a **single instance** with: diff --git a/doc/integration/advanced_search/elasticsearch.md b/doc/integration/advanced_search/elasticsearch.md index c123611fa44..5eac6ab7c84 100644 --- a/doc/integration/advanced_search/elasticsearch.md +++ b/doc/integration/advanced_search/elasticsearch.md @@ -95,11 +95,13 @@ The former Ruby-based indexer was removed in [GitLab 12.3](https://gitlab.com/gi First, we need to install some dependencies, then we build and install the indexer itself. +#### Install dependencies + This project relies on [International Components for Unicode](https://icu.unicode.org/) (ICU) for text encoding, therefore we must ensure the development packages for your platform are installed before running `make`. -#### Debian / Ubuntu +##### Debian / Ubuntu To install on Debian or Ubuntu, run: @@ -107,7 +109,7 @@ To install on Debian or Ubuntu, run: sudo apt install libicu-dev ``` -#### CentOS / RHEL +##### CentOS / RHEL To install on CentOS or RHEL, run: @@ -115,7 +117,7 @@ To install on CentOS or RHEL, run: sudo yum install libicu-devel ``` -#### macOS +##### macOS NOTE: You must first [install Homebrew](https://brew.sh/). @@ -127,7 +129,7 @@ brew install icu4c export PKG_CONFIG_PATH="/usr/local/opt/icu4c/lib/pkgconfig:$PKG_CONFIG_PATH" ``` -### Build and install +#### Build and install To build and install the indexer, run: @@ -294,7 +296,7 @@ To disable the Elasticsearch integration: bundle exec rake gitlab:elastic:delete_index RAILS_ENV=production ``` -## Unpause Indexing +## Unpause Indexing 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Settings > Advanced Search**. @@ -475,6 +477,7 @@ The following are some available Rake tasks: | Task | Description | |:--------------------------------------------------------------------------------------------------------------------------------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [`sudo gitlab-rake gitlab:elastic:info`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Outputs debugging information for the Advanced Search intergation. | | [`sudo gitlab-rake gitlab:elastic:index`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Enables Elasticsearch indexing and run `gitlab:elastic:create_empty_index`, `gitlab:elastic:clear_index_status`, `gitlab:elastic:index_projects`, and `gitlab:elastic:index_snippets`. | | [`sudo gitlab-rake gitlab:elastic:pause_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Pauses Elasticsearch indexing. Changes are still tracked. Useful for cluster/index migrations. | | [`sudo gitlab-rake gitlab:elastic:resume_indexing`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/tasks/gitlab/elastic.rake) | Resumes Elasticsearch indexing. | diff --git a/doc/integration/github.md b/doc/integration/github.md index 3c14e8db4de..3011155f825 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -218,4 +218,4 @@ To fix this issue, you must activate GitHub sign-in in GitLab: 1. On the top bar, in the top right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Account**. -1. In the **Social sign-in** section, select **Connect to GitHub**. +1. In the **Service sign-in** section, select **Connect to GitHub**. diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md index 68ba676b539..02705d9dec3 100644 --- a/doc/integration/gitlab.md +++ b/doc/integration/gitlab.md @@ -117,10 +117,9 @@ signed in. ## Reduce access privileges on sign in -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337663) in GitLab 14.8 [with a flag](../administration/feature_flags.md) named `omniauth_login_minimal_scopes`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `omniauth_login_minimal_scopes`. On GitLab.com, this feature is not available. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/337663) in GitLab 14.8 [with a flag](../administration/feature_flags.md) named `omniauth_login_minimal_scopes`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/351331) in GitLab 14.9. +> - [Feature flag `omniauth_login_minimal_scopes`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83453) removed in GitLab 15.2 If you use a GitLab instance for authentication, you can reduce access rights when an OAuth application is used for sign in. diff --git a/doc/integration/jira/development_panel.md b/doc/integration/jira/development_panel.md index 2f0ebea165a..4d5a9f92257 100644 --- a/doc/integration/jira/development_panel.md +++ b/doc/integration/jira/development_panel.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Jira development panel integration **(FREE)** -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/233149) from GitLab Premium to GitLab Free in 13.4. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/233149) from GitLab Premium to GitLab Free in 13.4. With the Jira development panel integration, you can reference Jira issues in GitLab. When configured, activity (such as pipeline, deployment, and feature flags) displays in the Jira issue's diff --git a/doc/integration/jira/index.md b/doc/integration/jira/index.md index 4e8b5ca8927..2f694094940 100644 --- a/doc/integration/jira/index.md +++ b/doc/integration/jira/index.md @@ -119,6 +119,6 @@ and complete the CAPTCHA. ### Jira integration does not work for imported project -There is a [known bug](https://gitlab.com/gitlab-org/gitlab/-/issues/341571) -where the Jira integration sometimes does not work for a project that has been imported. +There is a [known bug](https://gitlab.com/gitlab-org/gitlab/-/issues/341571) +where the Jira integration sometimes does not work for a project that has been imported. As a workaround, disable the integration and then re-enable it. diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md index 78c92631e4f..257ba4e6708 100644 --- a/doc/integration/kerberos.md +++ b/doc/integration/kerberos.md @@ -125,8 +125,8 @@ If you're not an administrator: 1. In the top-right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Account**. -1. In the **Social sign-in** section, select **Connect Kerberos SPNEGO**. - If you don't see a **Social sign-in** Kerberos option, follow the +1. In the **Service sign-in** section, select **Connect Kerberos SPNEGO**. + If you don't see a **Service sign-in** Kerberos option, follow the requirements in [Enable single sign-on](#enable-single-sign-on). In either case, you should now be able to sign in to your GitLab account diff --git a/doc/integration/oauth_provider.md b/doc/integration/oauth_provider.md index e33f874d35a..962f5c4e5fb 100644 --- a/doc/integration/oauth_provider.md +++ b/doc/integration/oauth_provider.md @@ -86,7 +86,7 @@ To create an application for your GitLab instance: When creating application in the **Admin Area** , you can mark it as _trusted_. The user authorization step is automatically skipped for this application. -## Expiring access tokens +## Access token expiration > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21745) in GitLab 14.3, with the ability to opt out. > - Ability to opt-out of expiring access token [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/340848) in GitLab 15.0. @@ -96,13 +96,8 @@ The ability to opt-out of expiring access tokens was [deprecated](https://gitlab in GitLab 14.3 and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/340848) in 15.0. All existing integrations must be updated to support access token refresh. -Access tokens expire in two hours which means that integrations that use them must support generating new access -tokens at least every two hours. Existing: - -- Applications can have expiring access tokens: - 1. Edit the application. - 1. Select **Expire access tokens**. -- Tokens must be [revoked](../api/oauth2.md#revoke-a-token) or they don't expire. +Access tokens expire after two hours. Integrations that use access tokens must generate new ones at least every +two hours. When applications are deleted, all grants and tokens associated with the application are also deleted. diff --git a/doc/integration/slash_commands.md b/doc/integration/slash_commands.md index f5916b72d9d..db1d5a8cce4 100644 --- a/doc/integration/slash_commands.md +++ b/doc/integration/slash_commands.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Slash commands in Mattermost and Slack **(FREE)** -> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24780) from GitLab Ultimate to GitLab Free in 11.9. +> [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24780) from GitLab Ultimate to GitLab Free in 11.9. If you want to control and view GitLab content while you're working in Slack and Mattermost, you can use slash commands. diff --git a/doc/integration/twitter.md b/doc/integration/twitter.md index 19eb31ec588..f3b66f8c12d 100644 --- a/doc/integration/twitter.md +++ b/doc/integration/twitter.md @@ -4,7 +4,10 @@ group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Twitter OAuth 2.0 OmniAuth Provider **(FREE SELF)** +# Twitter OAuth 1.0a OmniAuth Provider **(FREE SELF)** + +NOTE: +Twitter OAuth 2.0 support is [not yet supported](https://gitlab.com/gitlab-org/gitlab/-/issues/366213). To enable the Twitter OmniAuth provider you must register your application with Twitter. Twitter generates a client ID and secret key for you to use. diff --git a/doc/operations/feature_flags.md b/doc/operations/feature_flags.md index 71248adc777..9e7d452c259 100644 --- a/doc/operations/feature_flags.md +++ b/doc/operations/feature_flags.md @@ -297,7 +297,6 @@ For API content, see: - [Feature Flags API](../api/feature_flags.md) - [Feature Flag Specs API](../api/feature_flag_specs.md) (Deprecated and [scheduled for removal](https://gitlab.com/gitlab-org/gitlab/-/issues/213369) in GitLab 14.0.) - [Feature Flag User Lists API](../api/feature_flag_user_lists.md) -- [Legacy Feature Flags API](../api/feature_flags_legacy.md) ### Golang application example diff --git a/doc/operations/incident_management/alerts.md b/doc/operations/incident_management/alerts.md index af42571f82f..a4b34807094 100644 --- a/doc/operations/incident_management/alerts.md +++ b/doc/operations/incident_management/alerts.md @@ -168,8 +168,9 @@ by changing the status. Setting the status to: - **Acknowledged** limits on-call pages based on the project's [escalation policy](escalation_policies.md). - **Triggered** from **Resolved** restarts the alert escalating from the beginning. -For [alerts with an associated incident](alerts.md#create-an-incident-from-an-alert), -updating the alert status also updates the incident status. +In GitLab 15.1 and earlier, updating the status of an [alert with an associated incident](alerts.md#create-an-incident-from-an-alert) +also updates the incident status. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), +the incident status is independent and does not update when the alert status changes. ### Create an incident from an alert diff --git a/doc/operations/incident_management/img/timeline_events_v15_1.png b/doc/operations/incident_management/img/timeline_events_v15_1.png Binary files differnew file mode 100644 index 00000000000..3241f35764c --- /dev/null +++ b/doc/operations/incident_management/img/timeline_events_v15_1.png diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index e48127b3415..f1628cb64ca 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -53,7 +53,8 @@ GitLab to create incident automatically whenever an alert is triggered: 1. Navigate to **Settings > Monitor > Incidents** and expand **Incidents**. 1. Check the **Create an incident** checkbox. 1. To customize the incident, select an - [issue template](../../user/project/description_templates.md#create-an-issue-template). + [issue template](../../user/project/description_templates.md#create-an-issue-template), + to include in the [incident summary](#summary). 1. To send [an email notification](paging.md#email-notifications-for-alerts) to users with the Developer role, select **Send a separate email notification to Developers**. Email notifications are @@ -153,9 +154,9 @@ Select any incident in the list to display its incident details page. ### Summary -The summary section for incidents provides both critical details about and the -contents of the issue template (if one was used). The highlighted bar at the top -of the incident displays from left to right: +The summary section for incidents provides both critical details about the +incident and the contents of the issue template (if applicable). The highlighted +bar at the top of the incident displays from left to right: - The link to the original alert. - The alert start time. @@ -168,6 +169,13 @@ Beneath the highlight bar, GitLab displays a summary that includes the following - `full_query` - Monitoring tool +The incident summary can be further customized using +[GitLab Flavored Markdown](../../user/markdown.md). If the corresponding alert +[provided Markdown for the incident](../metrics/alerts.md#trigger-actions-from-alerts), +then the Markdown is appended to the summary after the above alert fields. If an +[incident template](#create-incidents-automatically) is configured for the +project, then the template content is appended at the end. + Comments are displayed in threads, but can be displayed chronologically [by toggling on the recent updates view](#recent-updates-view). @@ -195,6 +203,67 @@ field populated. ![Incident alert details](img/incident_alert_details_v13_4.png) +### Timeline events + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/344059) in GitLab 15.2 [with a flag](../../administration/feature_flags.md) named `incident_timeline`. Enabled on GitLab.com. Disabled on self-managed. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `incident_timeline`. +On GitLab.com, this feature is available. + +Incident timelines are an important part of record keeping for incidents. +They give a high-level overview, to executives and external viewers, of what happened during the incident, +and the steps that were taken for it to be resolved. + +#### View the event timeline + +Incident timeline events are listed in ascending order of the date and time. +They are grouped with dates and are listed in ascending order of the time when they occured: + +![Incident timeline events list](img/timeline_events_v15_1.png) + +To view the event timeline of an incident: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Monitor > Incidents**. +1. Select an incident. +1. Select the **Timeline** tab. + +#### Create a timeline event + +Create a timeline event manually using the form. + +Prerequisites: + +- You must have at least the Developer role for the project. + +To create a timeline event: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Monitor > Incidents**. +1. Select an incident. +1. Select the **Timeline** tab. +1. Select **Add new timeline event**. +1. Complete the required fields. +1. Select **Save** or **Save and add another event**. + +#### Delete a timeline event + +You can also delete timeline events. + +Prerequisites: + +- You must have at least the Developer role for the project. + +To delete a timeline event: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Monitor > Incidents**. +1. Select an incident. +1. Select the **Timeline** tab. +1. On the right of a timeline event, select **More actions** (**{ellipsis_v}**) and then select **Delete**. +1. To confirm, select **Delete Event**. + ### Recent updates view **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227836) in GitLab 13.5. @@ -270,8 +339,9 @@ by changing the status. Setting the status to: - **Acknowledged** limits on-call pages based on the selected [escalation policy](#change-escalation-policy). - **Triggered** from **Resolved** restarts the incident escalating from the beginning. -For [incidents created from alerts](alerts.md#create-an-incident-from-an-alert), -updating the incident status also updates the alert status. +In GitLab 15.1 and earlier, updating the status of an [incident created from an alert](alerts.md#create-an-incident-from-an-alert) +also updates the alert status. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), +the alert status is independent and does not update when the incident status changes. ### Change escalation policy **(PREMIUM)** @@ -288,8 +358,9 @@ Selecting an escalation policy updates the incident status to **Triggered** and Deselecting an escalation policy halts escalation. Refer to the [incident status](#change-incident-status) to manage on-call paging once escalation has begun. -For [incidents created from alerts](alerts.md#create-an-incident-from-an-alert), -the incident's escalation policy reflects the alert's escalation policy and cannot be changed. +In GitLab 15.1 and earlier, the escalation policy for [incidents created from alerts](alerts.md#create-an-incident-from-an-alert) +reflects the alert's escalation policy and cannot be changed. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), +the incident escalation policy is independent and can be changed. ### Manage incidents from Slack diff --git a/doc/operations/incident_management/paging.md b/doc/operations/incident_management/paging.md index 936c297e555..3eeeb67bf51 100644 --- a/doc/operations/incident_management/paging.md +++ b/doc/operations/incident_management/paging.md @@ -61,5 +61,6 @@ the rule fires. You can respond to a page or stop incident escalations by [updating the incident's status](incidents.md#change-incident-status) or, if applicable, [unsetting the incident's escalation policy](incidents.md#change-escalation-policy). -To avoid duplicate pages, [incidents created from alerts](alerts.md#create-an-incident-from-an-alert) do not support independent escalation. -Instead, the status and escalation policy fields are synced between the alert and the incident. +In GitLab 15.1 and earlier, [incidents created from alerts](alerts.md#create-an-incident-from-an-alert) +do not support independent escalation. In [GitLab 15.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/356057), +all incidents can be escalated independently. diff --git a/doc/operations/index.md b/doc/operations/index.md index 88687c2faf1..05ce1c5d876 100644 --- a/doc/operations/index.md +++ b/doc/operations/index.md @@ -77,29 +77,14 @@ microservices-based distributed systems - and displays results within GitLab. - [Trace the performance and health](tracing.md) of a deployed application. -## Aggregate and store logs (DEPRECATED) **(FREE SELF)** +<!--- start_remove The following content will be removed on remove_date: '2022-10-18'---> -> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346485) in GitLab 14.7. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/360182) behind a [feature flag](../administration/feature_flags.md) named `monitor_logging` in GitLab 15.0. Disabled by default. +## Aggregate and store logs (removed) **(FREE SELF)** -WARNING: -This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346485) -in GitLab 14.7. -It will be removed completely in GitLab 15.2. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `monitor_logging`. -On GitLab.com, this feature is not available. -This feature is not recommended for production use. - -Developers need to troubleshoot application changes in development, and incident -responders need aggregated, real-time logs when troubleshooting problems with -production services. GitLab provides centralized, aggregated log storage for your -distributed application, enabling you to collect logs across multiple services and -infrastructure. - -- [View logs of pods](../user/project/clusters/kubernetes_pod_logs.md) - in connected Kubernetes clusters. +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346485) in GitLab 14.7 +and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/360193) in GitLab 15.2. + +<!--- end_remove --> ## Manage your infrastructure in code diff --git a/doc/operations/metrics/dashboards/index.md b/doc/operations/metrics/dashboards/index.md index 152e3dca27d..bd1f75b7b42 100644 --- a/doc/operations/metrics/dashboards/index.md +++ b/doc/operations/metrics/dashboards/index.md @@ -135,9 +135,6 @@ The options are: - **Expand panel** - Displays a larger version of a visualization. To return to the dashboard, select the **Back** button in your browser, or press the <kbd>Escape</kbd> key. ([Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3100) in GitLab 13.0.) -- **View logs** **(ULTIMATE)** - Displays [Logs](../../../user/project/clusters/kubernetes_pod_logs.md), - if they are enabled. If used in conjunction with the [timeline zoom](#timeline-zoom-and-url-sharing) - feature, logs narrow down to the selected time range. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/122013) in GitLab 12.8.) - **Download CSV** - Data from Prometheus charts on the metrics dashboard can be downloaded as CSV. - [Copy link to chart](../embed.md#embedding-gitlab-managed-kubernetes-metrics) diff --git a/doc/operations/metrics/embed_grafana.md b/doc/operations/metrics/embed_grafana.md index 700bf821a49..2229ba22be7 100644 --- a/doc/operations/metrics/embed_grafana.md +++ b/doc/operations/metrics/embed_grafana.md @@ -54,6 +54,11 @@ To set up the Grafana API in Grafana: 1. For **API Token**, enter the Administrator API token you just generated. 1. Select **Save Changes**. +NOTE: +If the Grafana integration is enabled, any user with read access to the GitLab +project can query metrics from the Prometheus instance. All requests proxied +through GitLab are authenticated with the same Grafana Administrator API token. + ### Generate a link to a panel To generate a link to a panel: diff --git a/doc/operations/metrics/img/prometheus_integration_alerts.png b/doc/operations/metrics/img/prometheus_integration_alerts.png Binary files differdeleted file mode 100644 index 609c5e5196c..00000000000 --- a/doc/operations/metrics/img/prometheus_integration_alerts.png +++ /dev/null diff --git a/doc/operations/metrics/index.md b/doc/operations/metrics/index.md index 916fe6b0ea9..82f093cf432 100644 --- a/doc/operations/metrics/index.md +++ b/doc/operations/metrics/index.md @@ -117,7 +117,7 @@ dashboard is visible to authenticated and non-authenticated users. ## Adding custom metrics -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28527) from GitLab Premium to GitLab Free in 12.10. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/28527) from GitLab Premium to GitLab Free in 12.10. Custom metrics can be monitored by adding them on the monitoring dashboard page. After saving them, they display on the environment metrics dashboard provided that either: diff --git a/doc/operations/product_analytics.md b/doc/operations/product_analytics.md index a55cbe906a0..98ba6a9203c 100644 --- a/doc/operations/product_analytics.md +++ b/doc/operations/product_analytics.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/policy/alpha-beta-support.md b/doc/policy/alpha-beta-support.md index a2612e2bac7..45f14e4f9a2 100644 --- a/doc/policy/alpha-beta-support.md +++ b/doc/policy/alpha-beta-support.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Support for Alpha, Beta, Limited Availability, and Generally Available Features **(PREMIUM)** -Some GitLab features are released as [Alpha or Beta versions](https://about.gitlab.com/support/statement-of-support.html#alpha--beta-features) which are not fully supported. All other features are considered to be Generally Available (GA). +Some GitLab features are released as [Alpha or Beta versions](https://about.gitlab.com/support/statement-of-support/#alpha-beta-features) which are not fully supported. All other features are considered to be Generally Available (GA). ## Alpha Features @@ -32,7 +32,7 @@ Characteristics of beta features: - Features and functions are not likely to change. - Data loss is not likely. -Your Support Contract provides **commercially-reasonable effort** support for Beta features, with the expectation that issues will require extra time and assistance from development to troubleshoot. +Your Support Contract provides **commercially-reasonable effort** support for Beta features, with the expectation that issues require extra time and assistance from development to troubleshoot. ## Limited Availability (LA) diff --git a/doc/policy/maintenance.md b/doc/policy/maintenance.md index e3910115657..e831303988b 100644 --- a/doc/policy/maintenance.md +++ b/doc/policy/maintenance.md @@ -43,12 +43,12 @@ The following table describes the version types and their release cadence: ## Upgrade recommendations We encourage everyone to run the [latest stable release](https://about.gitlab.com/releases/categories/releases/) -to ensure that you can easily upgrade to the most secure and feature-rich GitLab experience. -To make sure you can easily run the most recent stable release, we are working +to ensure that you can upgrade to the most secure and feature-rich GitLab experience. +To make sure you can run the most recent stable release, we are working hard to keep the update process simple and reliable. If you are unable to follow our monthly release cycle, there are a couple of -cases you need to consider. Follow the +cases you must consider. Follow the [upgrade paths guide](../update/index.md#upgrade-paths) to safely upgrade between versions. @@ -106,7 +106,7 @@ Backporting to more than one stable release is normally reserved for [security r In some cases, however, we may need to backport *a bug fix* to more than one stable release, depending on the severity of the bug. -The decision on whether backporting a change will be performed is done at the discretion of the +The decision on whether backporting a change is performed is done at the discretion of the [current release managers](https://about.gitlab.com/community/release-managers/), similar to what is described in the [managing bugs](https://gitlab.com/gitlab-org/gitlab/-/blob/master/PROCESS.md#managing-bugs) process, based on *all* of the following: @@ -124,7 +124,7 @@ For instance, if we release `13.2.1` with a fix for a severe bug introduced in `13.0.0`, we could backport the fix to a new `13.0.x`, and `13.1.x` patch release. Note that [severity](../development/contributing/issue_workflow.md#severity-labels) 3 and lower -requests will be automatically turned down. +requests are automatically turned down. To request backporting to more than one stable release for consideration, raise an issue in the [release/tasks](https://gitlab.com/gitlab-org/release/tasks/-/issues/new?issuable_template=Backporting-request) issue tracker. diff --git a/doc/public_access/public_access.md b/doc/public_access/public_access.md deleted file mode 100644 index 8b665972918..00000000000 --- a/doc/public_access/public_access.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../user/public_access.md' -remove_date: '2022-06-18' ---- - -This document was moved to [another location](../user/public_access.md). - -<!-- This redirect file can be deleted after <2022-06-18>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/raketasks/backup_gitlab.md b/doc/raketasks/backup_gitlab.md new file mode 100644 index 00000000000..8d5ae14a043 --- /dev/null +++ b/doc/raketasks/backup_gitlab.md @@ -0,0 +1,908 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Back up GitLab + +GitLab provides a command line interface to back up your entire instance, +including: + +- Database +- Attachments +- Git repositories data +- CI/CD job output logs +- CI/CD job artifacts +- LFS objects +- Terraform states ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331806) in GitLab 14.7) +- Container Registry images +- GitLab Pages content +- Packages ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332006) in GitLab 14.7) +- Snippets +- [Group wikis](../user/project/wiki/group.md) + +Backups do not include: + +- [Mattermost data](https://docs.mattermost.com/administration/config-settings.html#file-storage) +- Redis (and thus Sidekiq jobs) + +WARNING: +GitLab does not back up any configuration files (`/etc/gitlab`), TLS keys and certificates, or system +files. You are highly advised to read about [storing configuration files](#storing-configuration-files). + +WARNING: +The backup command requires [additional parameters](backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer) when +your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster. + +Depending on your version of GitLab, use the following command if you installed +GitLab using the Omnibus package: + +- GitLab 12.2 or later: + + ```shell + sudo gitlab-backup create + ``` + +- GitLab 12.1 and earlier: + + ```shell + gitlab-rake gitlab:backup:create + ``` + +If you installed GitLab from source, use the following command: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production +``` + +If you're running GitLab from within a Docker container, run the backup from +the host, based on your installed version of GitLab: + +- GitLab 12.2 or later: + + ```shell + docker exec -t <container name> gitlab-backup create + ``` + +- GitLab 12.1 and earlier: + + ```shell + docker exec -t <container name> gitlab-rake gitlab:backup:create + ``` + +If you're using the [GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab) +on a Kubernetes cluster, you can run the backup task by using `kubectl` to run the `backup-utility` +script on the GitLab toolbox pod. For more details, see the +[charts backup documentation](https://docs.gitlab.com/charts/backup-restore/backup.html). + +Similar to the Kubernetes case, if you have scaled out your GitLab cluster to +use multiple application servers, you should pick a designated node (that isn't +auto-scaled away) for running the backup Rake task. Because the backup Rake +task is tightly coupled to the main Rails application, this is typically a node +on which you're also running Puma or Sidekiq. + +Example output: + +```plaintext +Dumping database tables: +- Dumping table events... [DONE] +- Dumping table issues... [DONE] +- Dumping table keys... [DONE] +- Dumping table merge_requests... [DONE] +- Dumping table milestones... [DONE] +- Dumping table namespaces... [DONE] +- Dumping table notes... [DONE] +- Dumping table projects... [DONE] +- Dumping table protected_branches... [DONE] +- Dumping table schema_migrations... [DONE] +- Dumping table services... [DONE] +- Dumping table snippets... [DONE] +- Dumping table taggings... [DONE] +- Dumping table tags... [DONE] +- Dumping table users... [DONE] +- Dumping table users_projects... [DONE] +- Dumping table web_hooks... [DONE] +- Dumping table wikis... [DONE] +Dumping repositories: +- Dumping repository abcd... [DONE] +Creating backup archive: $TIMESTAMP_gitlab_backup.tar [DONE] +Deleting tmp directories...[DONE] +Deleting old backups... [SKIPPING] +``` + +## Storing configuration files + +The [backup Rake task](#back-up-gitlab) GitLab provides does _not_ store your +configuration files. The primary reason for this is that your database contains +items including encrypted information for two-factor authentication and the +CI/CD _secure variables_. Storing encrypted information in the same location +as its key defeats the purpose of using encryption in the first place. + +WARNING: +The secrets file is essential to preserve your database encryption key. + +At the very **minimum**, you must back up: + +For Omnibus: + +- `/etc/gitlab/gitlab-secrets.json` +- `/etc/gitlab/gitlab.rb` + +For installation from source: + +- `/home/git/gitlab/config/secrets.yml` +- `/home/git/gitlab/config/gitlab.yml` + +For [Docker installations](https://docs.gitlab.com/omnibus/docker/), you must +back up the volume where the configuration files are stored. If you created +the GitLab container according to the documentation, it should be in the +`/srv/gitlab/config` directory. + +For [GitLab Helm chart installations](https://gitlab.com/gitlab-org/charts/gitlab) +on a Kubernetes cluster, you must follow the +[Back up the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#backup-the-secrets) +instructions. + +You may also want to back up any TLS keys and certificates (`/etc/gitlab/ssl`, `/etc/gitlab/trusted-certs`), and your +[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079) +to avoid man-in-the-middle attack warnings if you have to perform a full machine restore. + +If you use Omnibus GitLab, review additional information to +[backup your configuration](https://docs.gitlab.com/omnibus/settings/backups.html). + +In the unlikely event that the secrets file is lost, see the +[troubleshooting section](backup_restore.md#when-the-secrets-file-is-lost). + +## Backup options + +The command line tool GitLab provides to backup your instance can accept more +options. + +### Backup strategy option + +The default backup strategy is to essentially stream data from the respective +data locations to the backup using the Linux command `tar` and `gzip`. This works +fine in most cases, but can cause problems when data is rapidly changing. + +When data changes while `tar` is reading it, the error `file changed as we read +it` may occur, and causes the backup process to fail. To combat this, 8.17 +introduces a new backup strategy called `copy`. The strategy copies data files +to a temporary location before calling `tar` and `gzip`, avoiding the error. + +A side-effect is that the backup process takes up to an additional 1X disk +space. The process does its best to clean up the temporary files at each stage +so the problem doesn't compound, but it could be a considerable change for large +installations. This is why the `copy` strategy is not the default in 8.17. + +To use the `copy` strategy instead of the default streaming strategy, specify +`STRATEGY=copy` in the Rake task command. For example: + +```shell +sudo gitlab-backup create STRATEGY=copy +``` + +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. + +### Backup filename + +WARNING: +If you use a custom backup filename, you can't +[limit the lifetime of the backups](#limit-backup-lifetime-for-local-files-prune-old-backups). + +By default, a backup file is created according to the specification in the +previous [Backup timestamp](backup_restore.md#backup-timestamp) section. You can, however, +override the `[TIMESTAMP]` portion of the filename by setting the `BACKUP` +environment variable. For example: + +```shell +sudo gitlab-backup create BACKUP=dump +``` + +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. + +The resulting file is named `dump_gitlab_backup.tar`. This is useful for +systems that make use of rsync and incremental backups, and results in +considerably faster transfer speeds. + +### Confirm archive can be transferred + +To ensure the generated archive is transferable by rsync, you can set the `GZIP_RSYNCABLE=yes` +option. This sets the `--rsyncable` option to `gzip`, which is useful only in +combination with setting [the Backup filename option](#backup-filename). + +Note that the `--rsyncable` option in `gzip` isn't guaranteed to be available +on all distributions. To verify that it's available in your distribution, run +`gzip --help` or consult the man pages. + +```shell +sudo gitlab-backup create BACKUP=dump GZIP_RSYNCABLE=yes +``` + +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. + +### Excluding specific directories from the backup + +You can exclude specific directories from the backup by adding the environment variable `SKIP`, whose values are a comma-separated list of the following options: + +- `db` (database) +- `uploads` (attachments) +- `builds` (CI job output logs) +- `artifacts` (CI job artifacts) +- `lfs` (LFS objects) +- `terraform_state` (Terraform states) +- `registry` (Container Registry images) +- `pages` (Pages content) +- `repositories` (Git repositories data) +- `packages` (Packages) + +All wikis are backed up as part of the `repositories` group. Non-existent wikis are skipped during a backup. + +NOTE: +When [backing up and restoring Helm Charts](https://docs.gitlab.com/charts/architecture/backup-restore.html), there is an additional option `packages`, which refers to any packages managed by the GitLab [package registry](../user/packages/package_registry/index.md). +For more information see [command line arguments](https://docs.gitlab.com/charts/architecture/backup-restore.html#command-line-arguments). + +All wikis are backed up as part of the `repositories` group. Non-existent +wikis are skipped during a backup. + +For Omnibus GitLab packages: + +```shell +sudo gitlab-backup create SKIP=db,uploads +``` + +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. + +For installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production +``` + +### Skipping tar creation + +NOTE: +It is not possible to skip the tar creation when using [object storage](#uploading-backups-to-a-remote-cloud-storage) for backups. + +The last part of creating a backup is generation of a `.tar` file containing +all the parts. In some cases (for example, if the backup is picked up by other +backup software) creating a `.tar` file might be wasted effort or even directly +harmful, so you can skip this step by adding `tar` to the `SKIP` environment +variable. + +Adding `tar` to the `SKIP` variable leaves the files and directories containing the +backup in the directory used for the intermediate files. These files are +overwritten when a new backup is created, so you should make sure they are copied +elsewhere, because you can only have one backup on the system. + +For Omnibus GitLab packages: + +```shell +sudo gitlab-backup create SKIP=tar +``` + +For installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create SKIP=tar RAILS_ENV=production +``` + +### Back up Git repositories concurrently + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37158) in GitLab 13.3. +> - [Concurrent restore introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69330) in GitLab 14.3 + +When using [multiple repository storages](../administration/repository_storage_paths.md), +repositories can be backed up or restored concurrently to help fully use CPU time. The +following variables are available to modify the default behavior of the Rake +task: + +- `GITLAB_BACKUP_MAX_CONCURRENCY`: The maximum number of projects to back up at + the same time. Defaults to the number of logical CPUs (in GitLab 14.1 and + earlier, defaults to `1`). +- `GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY`: The maximum number of projects to + back up at the same time on each storage. This allows the repository backups + to be spread across storages. Defaults to `2` (in GitLab 14.1 and earlier, + defaults to `1`). + +For example, for Omnibus GitLab installations with 4 repository storages: + +```shell +sudo gitlab-backup create GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1 +``` + +For example, for installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1 +``` + +### Incremental repository backups + +> - Introduced in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `incremental_repository_backup`. Disabled by default. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/355945) in GitLab 14.10. +> - `PREVIOUS_BACKUP` option [introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4184) in GitLab 15.0. + +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `incremental_repository_backup`. +On GitLab.com, this feature is not available. + +Incremental backups can be faster than full backups because they only pack changes since the last backup into the backup +bundle for each repository. There must be an existing backup to create an incremental backup from: + +- In GitLab 14.9 and 14.10, use the `BACKUP=<timestamp_of_backup>` option to choose the backup to use. The chosen previous backup is overwritten. +- In GitLab 15.0 and later, use the `PREVIOUS_BACKUP=<timestamp_of_backup>` option to choose the backup to use. By default, a backup file is created + as documented in the [Backup timestamp](backup_restore.md#backup-timestamp) section. You can override the `[TIMESTAMP]` portion of the filename by setting the + [`BACKUP` environment variable](#backup-filename). + +To create an incremental backup, run: + +```shell +sudo gitlab-backup create INCREMENTAL=yes PREVIOUS_BACKUP=<timestamp_of_backup> +``` + +Incremental backups can also be created from [an untarred backup](#skipping-tar-creation) by using `SKIP=tar`: + +```shell +sudo gitlab-backup create INCREMENTAL=yes SKIP=tar +``` + +### Back up specific repository storages + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86896) in GitLab 15.0. + +When using [multiple repository storages](../administration/repository_storage_paths.md), +repositories from specific repository storages can be backed up separately +using the `REPOSITORIES_STORAGES` option. The option accepts a comma-separated list of +storage names. + +For example, for Omnibus GitLab installations: + +```shell +sudo gitlab-backup create REPOSITORIES_STORAGES=storage1,storage2 +``` + +For example, for installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create REPOSITORIES_STORAGES=storage1,storage2 +``` + +### Back up specific repositories + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88094) in GitLab 15.1. + +You can back up a specific repositories using the `REPOSITORIES_PATHS` option. +The option accepts a comma-separated list of project and group paths. If you +specify a group path, all repositories in all projects in the group and +descendent groups are included. + +For example, to back up all repositories for all projects in **Group A** (`group-a`), and the repository for **Project C** in **Group B** (`group-b/project-c`): + +- Omnibus GitLab installations: + + ```shell + sudo gitlab-backup create REPOSITORIES_PATHS=group-a,group-b/project-c + ``` + +- Installations from source: + + ```shell + sudo -u git -H bundle exec rake gitlab:backup:create REPOSITORIES_PATHS=group-a,group-b/project-c + ``` + +### Uploading backups to a remote (cloud) storage + +NOTE: +It is not possible to [skip the tar creation](#skipping-tar-creation) when using object storage for backups. + +You can let the backup script upload (using the [Fog library](https://fog.io/)) +the `.tar` file it creates. In the following example, we use Amazon S3 for +storage, but Fog also lets you use [other storage providers](https://fog.io/storage/). +GitLab also [imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/-/blob/da46c9655962df7d49caef0e2b9f6bbe88462a02/Gemfile#L113) +for AWS, Google, OpenStack Swift, Rackspace, and Aliyun. A local driver is +[also available](#uploading-to-locally-mounted-shares). + +[Read more about using object storage with GitLab](../administration/object_storage.md). + +#### Using Amazon S3 + +For Omnibus GitLab packages: + +1. Add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'eu-west-1', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123' + # If using an IAM Profile, don't configure aws_access_key_id & aws_secret_access_key + # 'use_iam_profile' => true + } + gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' + ``` + +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect + +#### S3 Encrypted Buckets + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64765) in GitLab 14.3. + +AWS supports these [modes for server side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html): + +- Amazon S3-Managed Keys (SSE-S3) +- Customer Master Keys (CMKs) Stored in AWS Key Management Service (SSE-KMS) +- Customer-Provided Keys (SSE-C) + +Use your mode of choice with GitLab. Each mode has similar, but slightly +different, configuration methods. + +##### SSE-S3 + +To enable SSE-S3, in the backup storage options set the `server_side_encryption` +field to `AES256`. For example, in Omnibus GitLab: + +```ruby +gitlab_rails['backup_upload_storage_options'] = { + 'server_side_encryption' => 'AES256' +} +``` + +##### SSE-KMS + +To enable SSE-KMS, you'll need the [KMS key via its Amazon Resource Name (ARN) +in the `arn:aws:kms:region:acct-id:key/key-id` format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). Under the `backup_upload_storage_options` configuration setting, set: + +- `server_side_encryption` to `aws:kms`. +- `server_side_encryption_kms_key_id` to the ARN of the key. + +For example, in Omnibus GitLab: + +```ruby +gitlab_rails['backup_upload_storage_options'] = { + 'server_side_encryption' => 'aws:kms', + 'server_side_encryption_kms_key_id' => 'arn:aws:<YOUR KMS KEY ID>:' +} +``` + +##### SSE-C + +SSE-C requires you to set these encryption options: + +- `backup_encryption`: AES256. +- `backup_encryption_key`: Unencoded, 32-byte (256 bits) key. The upload fails if this isn't exactly 32 bytes. + +For example, in Omnibus GitLab: + +```ruby +gitlab_rails['backup_encryption'] = 'AES256' +gitlab_rails['backup_encryption_key'] = '<YOUR 32-BYTE KEY HERE>' +``` + +If the key contains binary characters and cannot be encoded in UTF-8, +instead, specify the key with the `GITLAB_BACKUP_ENCRYPTION_KEY` environment variable. +For example: + +```ruby +gitlab_rails['env'] = { 'GITLAB_BACKUP_ENCRYPTION_KEY' => "\xDE\xAD\xBE\xEF" * 8 } +``` + +#### Digital Ocean Spaces + +This example can be used for a bucket in Amsterdam (AMS3): + +1. Add the following to `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AWS', + 'region' => 'ams3', + 'aws_access_key_id' => 'AKIAKIAKI', + 'aws_secret_access_key' => 'secret123', + 'endpoint' => 'https://ams3.digitaloceanspaces.com' + } + gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' + ``` + +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect + +If you see a `400 Bad Request` error message when using Digital Ocean Spaces, +the cause may be the use of backup encryption. Because Digital Ocean Spaces +doesn't support encryption, remove or comment the line that contains +`gitlab_rails['backup_encryption']`. + +#### Other S3 Providers + +Not all S3 providers are fully compatible with the Fog library. For example, +if you see a `411 Length Required` error message after attempting to upload, +you may need to downgrade the `aws_signature_version` value from the default +value to `2`, [due to this issue](https://github.com/fog/fog-aws/issues/428). + +For installations from source: + +1. Edit `home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + # snip + upload: + # Fog storage connection settings, see https://fog.io/storage/ . + connection: + provider: AWS + region: eu-west-1 + aws_access_key_id: AKIAKIAKI + aws_secret_access_key: 'secret123' + # If using an IAM Profile, leave aws_access_key_id & aws_secret_access_key empty + # ie. aws_access_key_id: '' + # use_iam_profile: 'true' + # The remote 'directory' to store your backups. For S3, this would be the bucket name. + remote_directory: 'my.s3.bucket' + # Specifies Amazon S3 storage class to use for backups, this is optional + # storage_class: 'STANDARD' + # + # Turns on AWS Server-Side Encryption with Amazon Customer-Provided Encryption Keys for backups, this is optional + # 'encryption' must be set in order for this to have any effect. + # 'encryption_key' should be set to the 256-bit encryption key for Amazon S3 to use to encrypt or decrypt. + # To avoid storing the key on disk, the key can also be specified via the `GITLAB_BACKUP_ENCRYPTION_KEY` your data. + # encryption: 'AES256' + # encryption_key: '<key>' + # + # + # Turns on AWS Server-Side Encryption with Amazon S3-Managed keys (optional) + # https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html + # For SSE-S3, set 'server_side_encryption' to 'AES256'. + # For SS3-KMS, set 'server_side_encryption' to 'aws:kms'. Set + # 'server_side_encryption_kms_key_id' to the ARN of customer master key. + # storage_options: + # server_side_encryption: 'aws:kms' + # server_side_encryption_kms_key_id: 'arn:aws:kms:YOUR-KEY-ID-HERE' + ``` + +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect + +If you're uploading your backups to S3, you should create a new +IAM user with restricted access rights. To give the upload user access only for +uploading backups create the following IAM profile, replacing `my.s3.bucket` +with the name of your bucket: + +```json +{ + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "Stmt1412062044000", + "Effect": "Allow", + "Action": [ + "s3:AbortMultipartUpload", + "s3:GetBucketAcl", + "s3:GetBucketLocation", + "s3:GetObject", + "s3:GetObjectAcl", + "s3:ListBucketMultipartUploads", + "s3:PutObject", + "s3:PutObjectAcl" + ], + "Resource": [ + "arn:aws:s3:::my.s3.bucket/*" + ] + }, + { + "Sid": "Stmt1412062097000", + "Effect": "Allow", + "Action": [ + "s3:GetBucketLocation", + "s3:ListAllMyBuckets" + ], + "Resource": [ + "*" + ] + }, + { + "Sid": "Stmt1412062128000", + "Effect": "Allow", + "Action": [ + "s3:ListBucket" + ], + "Resource": [ + "arn:aws:s3:::my.s3.bucket" + ] + } + ] +} +``` + +#### Using Google Cloud Storage + +To use Google Cloud Storage to save backups, you must first create an +access key from the Google console: + +1. Go to the [Google storage settings page](https://console.cloud.google.com/storage/settings). +1. Select **Interoperability**, and then create an access key. +1. Make note of the **Access Key** and **Secret** and replace them in the + following configurations. +1. In the buckets advanced settings ensure the Access Control option + **Set object-level and bucket-level permissions** is selected. +1. Ensure you have already created a bucket. + +For Omnibus GitLab packages: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'Google', + 'google_storage_access_key_id' => 'Access Key', + 'google_storage_secret_access_key' => 'Secret', + + ## If you have CNAME buckets (foo.example.com), you might run into SSL issues + ## when uploading backups ("hostname foo.example.com.storage.googleapis.com + ## does not match the server certificate"). In that case, uncomnent the following + ## setting. See: https://github.com/fog/fog/issues/2834 + #'path_style' => true + } + gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket' + ``` + +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect + +For installations from source: + +1. Edit `home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + upload: + connection: + provider: 'Google' + google_storage_access_key_id: 'Access Key' + google_storage_secret_access_key: 'Secret' + remote_directory: 'my.google.bucket' + ``` + +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect + +#### Using Azure Blob storage + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25877) in GitLab 13.4. + +For Omnibus GitLab packages: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_upload_connection'] = { + 'provider' => 'AzureRM', + 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>', + 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>', + 'azure_storage_domain' => 'blob.core.windows.net', # Optional + } + gitlab_rails['backup_upload_remote_directory'] = '<AZURE BLOB CONTAINER>' + ``` + +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect + +For installations from source: + +1. Edit `home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + upload: + connection: + provider: 'AzureRM' + azure_storage_account_name: '<AZURE STORAGE ACCOUNT NAME>' + azure_storage_access_key: '<AZURE STORAGE ACCESS KEY>' + remote_directory: '<AZURE BLOB CONTAINER>' + ``` + +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect + +For more details, see the [table of Azure parameters](../administration/object_storage.md#azure-blob-storage). + +#### Specifying a custom directory for backups + +This option works only for remote storage. If you want to group your backups, +you can pass a `DIRECTORY` environment variable: + +```shell +sudo gitlab-backup create DIRECTORY=daily +sudo gitlab-backup create DIRECTORY=weekly +``` + +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. + +### Skip uploading backups to remote storage + +If you have configured GitLab to [upload backups in a remote storage](#uploading-backups-to-a-remote-cloud-storage), +you can use the `SKIP=remote` option to skip uploading your backups to the remote storage. + +For Omnibus GitLab packages: + +```shell +sudo gitlab-backup create SKIP=remote +``` + +For installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:create SKIP=remote RAILS_ENV=production +``` + +### Uploading to locally mounted shares + +You may also send backups to a mounted share (for example, `NFS`,`CIFS`, or +`SMB`) by using the Fog [`Local`](https://github.com/fog/fog-local#usage) +storage provider. The directory pointed to by the `local_root` key _must_ be +owned by the `git` user _when mounted_ (mounting with the `uid=` of the `git` +user for `CIFS` and `SMB`) or the user that you are executing the backup tasks +as (for Omnibus packages, this is the `git` user). + +The `backup_upload_remote_directory` _must_ be set in addition to the +`local_root` key. This is the sub directory inside the mounted directory that +backups are copied to, and is created if it does not exist. If the +directory that you want to copy the tarballs to is the root of your mounted +directory, use `.` instead. + +Because file system performance may affect overall GitLab performance, +[GitLab doesn't recommend using cloud-based file systems for storage](../administration/nfs.md#avoid-using-cloud-based-file-systems). + +For Omnibus GitLab packages: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_upload_connection'] = { + :provider => 'Local', + :local_root => '/mnt/backups' + } + + # The directory inside the mounted folder to copy backups to + # Use '.' to store them in the root directory + gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups' + ``` + +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +For installations from source: + +1. Edit `home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + upload: + # Fog storage connection settings, see https://fog.io/storage/ . + connection: + provider: Local + local_root: '/mnt/backups' + # The directory inside the mounted folder to copy backups to + # Use '.' to store them in the root directory + remote_directory: 'gitlab_backups' + ``` + +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect. + +### Backup archive permissions + +The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`) +have the owner/group `git`/`git` and 0600 permissions by default. This is +meant to avoid other system users reading GitLab data. If you need the backup +archives to have different permissions, you can use the `archive_permissions` +setting. + +For Omnibus GitLab packages: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable + ``` + +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +For installations from source: + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + archive_permissions: 0644 # Makes the backup archives world-readable + ``` + +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect. + +### Configuring cron to make daily backups + +WARNING: +The following cron jobs do not [back up your GitLab configuration files](#storing-configuration-files) +or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). + +You can schedule a cron job that backs up your repositories and GitLab metadata. + +For Omnibus GitLab packages: + +1. Edit the crontab for the `root` user: + + ```shell + sudo su - + crontab -e + ``` + +1. There, add the following line to schedule the backup for everyday at 2 AM: + + ```plaintext + 0 2 * * * /opt/gitlab/bin/gitlab-backup create CRON=1 + ``` + + Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. + +For installations from source: + +1. Edit the crontab for the `git` user: + + ```shell + sudo -u git crontab -e + ``` + +1. Add the following lines at the bottom: + + ```plaintext + # Create a full backup of the GitLab repositories and SQL database every day at 2am + 0 2 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1 + ``` + +The `CRON=1` environment setting directs the backup script to hide all progress +output if there aren't any errors. This is recommended to reduce cron spam. +When troubleshooting backup problems, however, replace `CRON=1` with `--trace` to log verbosely. + +## Limit backup lifetime for local files (prune old backups) + +WARNING: +The process described in this section don't work if you used a [custom filename](#backup-filename) +for your backups. + +To prevent regular backups from using all your disk space, you may want to set a limited lifetime +for backups. The next time the backup task runs, backups older than the `backup_keep_time` are +pruned. + +This configuration option manages only local files. GitLab doesn't prune old +files stored in a third-party [object storage](#uploading-backups-to-a-remote-cloud-storage) +because the user may not have permission to list and delete files. It's +recommended that you configure the appropriate retention policy for your object +storage (for example, [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html)). + +For Omnibus GitLab packages: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + ## Limit backup lifetime to 7 days - 604800 seconds + gitlab_rails['backup_keep_time'] = 604800 + ``` + +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) + for the changes to take effect. + +For installations from source: + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + backup: + ## Limit backup lifetime to 7 days - 604800 seconds + keep_time: 604800 + ``` + +1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) + for the changes to take effect. diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index d14263f660a..33917ca9410 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -25,8 +25,8 @@ backups with your object storage provider, if desired. To be able to back up and restore, ensure that Rsync is installed on your system. If you installed GitLab: -- _Using the Omnibus package_, you're all set. -- _From source_, you need to determine if `rsync` is installed. For example: +- _Using the Omnibus package_, Rsync is already installed. +- _From source_, check if `rsync` is installed. If Rsync is not installed, install it. For example: ```shell # Debian/Ubuntu @@ -36,1281 +36,46 @@ system. If you installed GitLab: sudo yum install rsync ``` -## Backup timestamp - -The backup archive is saved in `backup_path`, which is specified in the -`config/gitlab.yml` file. The filename is `[TIMESTAMP]_gitlab_backup.tar`, -where `TIMESTAMP` identifies the time at which each backup was created, plus -the GitLab version. The timestamp is needed if you need to restore GitLab and -multiple backups are available. - -For example, if the backup name is `1493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar`, -the timestamp is `1493107454_2018_04_25_10.6.4-ce`. - -## Back up GitLab - -GitLab provides a command line interface to back up your entire instance, -including: - -- Database -- Attachments -- Git repositories data -- CI/CD job output logs -- CI/CD job artifacts -- LFS objects -- Terraform states ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331806) in GitLab 14.7) -- Container Registry images -- GitLab Pages content -- Packages ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332006) in GitLab 14.7) -- Snippets -- [Group wikis](../user/project/wiki/group.md) - -Backups do not include: - -- [Mattermost data](https://docs.mattermost.com/administration/config-settings.html#file-storage) -- Redis (and thus Sidekiq jobs) - -WARNING: -GitLab does not back up any configuration files (`/etc/gitlab`), TLS keys and certificates, or system -files. You are highly advised to read about [storing configuration files](#storing-configuration-files). - -WARNING: -The backup command requires [additional parameters](#back-up-and-restore-for-installations-using-pgbouncer) when -your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster. - -Depending on your version of GitLab, use the following command if you installed -GitLab using the Omnibus package: - -- GitLab 12.2 or later: - - ```shell - sudo gitlab-backup create - ``` - -- GitLab 12.1 and earlier: - - ```shell - gitlab-rake gitlab:backup:create - ``` - -If you installed GitLab from source, use the following command: - -```shell -sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production -``` - -If you're running GitLab from within a Docker container, run the backup from -the host, based on your installed version of GitLab: - -- GitLab 12.2 or later: - - ```shell - docker exec -t <container name> gitlab-backup create - ``` - -- GitLab 12.1 and earlier: - - ```shell - docker exec -t <container name> gitlab-rake gitlab:backup:create - ``` - -If you're using the [GitLab Helm chart](https://gitlab.com/gitlab-org/charts/gitlab) -on a Kubernetes cluster, you can run the backup task by using `kubectl` to run the `backup-utility` -script on the GitLab toolbox pod. For more details, see the -[charts backup documentation](https://docs.gitlab.com/charts/backup-restore/backup.html). - -Similar to the Kubernetes case, if you have scaled out your GitLab cluster to -use multiple application servers, you should pick a designated node (that isn't -auto-scaled away) for running the backup Rake task. Because the backup Rake -task is tightly coupled to the main Rails application, this is typically a node -on which you're also running Puma or Sidekiq. - -Example output: - -```plaintext -Dumping database tables: -- Dumping table events... [DONE] -- Dumping table issues... [DONE] -- Dumping table keys... [DONE] -- Dumping table merge_requests... [DONE] -- Dumping table milestones... [DONE] -- Dumping table namespaces... [DONE] -- Dumping table notes... [DONE] -- Dumping table projects... [DONE] -- Dumping table protected_branches... [DONE] -- Dumping table schema_migrations... [DONE] -- Dumping table services... [DONE] -- Dumping table snippets... [DONE] -- Dumping table taggings... [DONE] -- Dumping table tags... [DONE] -- Dumping table users... [DONE] -- Dumping table users_projects... [DONE] -- Dumping table web_hooks... [DONE] -- Dumping table wikis... [DONE] -Dumping repositories: -- Dumping repository abcd... [DONE] -Creating backup archive: $TIMESTAMP_gitlab_backup.tar [DONE] -Deleting tmp directories...[DONE] -Deleting old backups... [SKIPPING] -``` - -### Storing configuration files - -The [backup Rake task](#back-up-gitlab) GitLab provides does _not_ store your -configuration files. The primary reason for this is that your database contains -items including encrypted information for two-factor authentication and the -CI/CD _secure variables_. Storing encrypted information in the same location -as its key defeats the purpose of using encryption in the first place. - -WARNING: -The secrets file is essential to preserve your database encryption key. - -At the very **minimum**, you must back up: - -For Omnibus: - -- `/etc/gitlab/gitlab-secrets.json` -- `/etc/gitlab/gitlab.rb` - -For installation from source: - -- `/home/git/gitlab/config/secrets.yml` -- `/home/git/gitlab/config/gitlab.yml` - -For [Docker installations](https://docs.gitlab.com/omnibus/docker/), you must -back up the volume where the configuration files are stored. If you created -the GitLab container according to the documentation, it should be in the -`/srv/gitlab/config` directory. - -For [GitLab Helm chart installations](https://gitlab.com/gitlab-org/charts/gitlab) -on a Kubernetes cluster, you must follow the -[Back up the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#backup-the-secrets) -instructions. - -You may also want to back up any TLS keys and certificates (`/etc/gitlab/ssl`, `/etc/gitlab/trusted-certs`), and your -[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079) -to avoid man-in-the-middle attack warnings if you have to perform a full machine restore. - -If you use Omnibus GitLab, review additional information to -[backup your configuration](https://docs.gitlab.com/omnibus/settings/backups.html). - -In the unlikely event that the secrets file is lost, see the -[troubleshooting section](#when-the-secrets-file-is-lost). - -### Backup options - -The command line tool GitLab provides to backup your instance can accept more -options. - -#### Backup strategy option - -The default backup strategy is to essentially stream data from the respective -data locations to the backup using the Linux command `tar` and `gzip`. This works -fine in most cases, but can cause problems when data is rapidly changing. - -When data changes while `tar` is reading it, the error `file changed as we read -it` may occur, and causes the backup process to fail. To combat this, 8.17 -introduces a new backup strategy called `copy`. The strategy copies data files -to a temporary location before calling `tar` and `gzip`, avoiding the error. - -A side-effect is that the backup process takes up to an additional 1X disk -space. The process does its best to clean up the temporary files at each stage -so the problem doesn't compound, but it could be a considerable change for large -installations. This is why the `copy` strategy is not the default in 8.17. - -To use the `copy` strategy instead of the default streaming strategy, specify -`STRATEGY=copy` in the Rake task command. For example: - -```shell -sudo gitlab-backup create STRATEGY=copy -``` - -Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. - -#### Backup filename - -WARNING: -If you use a custom backup filename, you can't -[limit the lifetime of the backups](#limit-backup-lifetime-for-local-files-prune-old-backups). - -By default, a backup file is created according to the specification in the -previous [Backup timestamp](#backup-timestamp) section. You can, however, -override the `[TIMESTAMP]` portion of the filename by setting the `BACKUP` -environment variable. For example: - -```shell -sudo gitlab-backup create BACKUP=dump -``` - -Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. - -The resulting file is named `dump_gitlab_backup.tar`. This is useful for -systems that make use of rsync and incremental backups, and results in -considerably faster transfer speeds. - -#### Confirm archive can be transferred - -To ensure the generated archive is transferable by rsync, you can set the `GZIP_RSYNCABLE=yes` -option. This sets the `--rsyncable` option to `gzip`, which is useful only in -combination with setting [the Backup filename option](#backup-filename). - -Note that the `--rsyncable` option in `gzip` isn't guaranteed to be available -on all distributions. To verify that it's available in your distribution, run -`gzip --help` or consult the man pages. - -```shell -sudo gitlab-backup create BACKUP=dump GZIP_RSYNCABLE=yes -``` - -Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. - -#### Excluding specific directories from the backup - -You can exclude specific directories from the backup by adding the environment variable `SKIP`, whose values are a comma-separated list of the following options: - -- `db` (database) -- `uploads` (attachments) -- `builds` (CI job output logs) -- `artifacts` (CI job artifacts) -- `lfs` (LFS objects) -- `terraform_state` (Terraform states) -- `registry` (Container Registry images) -- `pages` (Pages content) -- `repositories` (Git repositories data) -- `packages` (Packages) - -All wikis are backed up as part of the `repositories` group. Non-existent wikis are skipped during a backup. - -NOTE: -When [backing up and restoring Helm Charts](https://docs.gitlab.com/charts/architecture/backup-restore.html), there is an additional option `packages`, which refers to any packages managed by the GitLab [package registry](../user/packages/package_registry/index.md). -For more information see [command line arguments](https://docs.gitlab.com/charts/architecture/backup-restore.html#command-line-arguments). - -All wikis are backed up as part of the `repositories` group. Non-existent -wikis are skipped during a backup. - -For Omnibus GitLab packages: - -```shell -sudo gitlab-backup create SKIP=db,uploads -``` - -Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. - -For installations from source: - -```shell -sudo -u git -H bundle exec rake gitlab:backup:create SKIP=db,uploads RAILS_ENV=production -``` - -#### Skipping tar creation - -NOTE: -It is not possible to skip the tar creation when using [object storage](#uploading-backups-to-a-remote-cloud-storage) for backups. - -The last part of creating a backup is generation of a `.tar` file containing -all the parts. In some cases (for example, if the backup is picked up by other -backup software) creating a `.tar` file might be wasted effort or even directly -harmful, so you can skip this step by adding `tar` to the `SKIP` environment -variable. - -Adding `tar` to the `SKIP` variable leaves the files and directories containing the -backup in the directory used for the intermediate files. These files are -overwritten when a new backup is created, so you should make sure they are copied -elsewhere, because you can only have one backup on the system. - -For Omnibus GitLab packages: - -```shell -sudo gitlab-backup create SKIP=tar -``` - -For installations from source: - -```shell -sudo -u git -H bundle exec rake gitlab:backup:create SKIP=tar RAILS_ENV=production -``` - -#### Disabling prompts during restore - -During a restore from backup, the restore script may ask for confirmation before -proceeding. If you wish to disable these prompts, you can set the `GITLAB_ASSUME_YES` -environment variable to `1`. - -For Omnibus GitLab packages: - -```shell -sudo GITLAB_ASSUME_YES=1 gitlab-backup restore -``` - -For installations from source: - -```shell -sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_ENV=production -``` - -#### Back up Git repositories concurrently - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37158) in GitLab 13.3. -> - [Concurrent restore introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69330) in GitLab 14.3 - -When using [multiple repository storages](../administration/repository_storage_paths.md), -repositories can be backed up or restored concurrently to help fully use CPU time. The -following variables are available to modify the default behavior of the Rake -task: - -- `GITLAB_BACKUP_MAX_CONCURRENCY`: The maximum number of projects to back up at - the same time. Defaults to the number of logical CPUs (in GitLab 14.1 and - earlier, defaults to `1`). -- `GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY`: The maximum number of projects to - back up at the same time on each storage. This allows the repository backups - to be spread across storages. Defaults to `2` (in GitLab 14.1 and earlier, - defaults to `1`). - -For example, for Omnibus GitLab installations with 4 repository storages: - -```shell -sudo gitlab-backup create GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1 -``` - -For example, for installations from source: - -```shell -sudo -u git -H bundle exec rake gitlab:backup:create GITLAB_BACKUP_MAX_CONCURRENCY=4 GITLAB_BACKUP_MAX_STORAGE_CONCURRENCY=1 -``` - -#### Incremental repository backups - -> - Introduced in GitLab 14.9 [with a flag](../administration/feature_flags.md) named `incremental_repository_backup`. Disabled by default. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/355945) in GitLab 14.10. -> - `PREVIOUS_BACKUP` option [introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/4184) in GitLab 15.0. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `incremental_repository_backup`. -On GitLab.com, this feature is not available. - -Incremental backups can be faster than full backups because they only pack changes since the last backup into the backup -bundle for each repository. There must be an existing backup to create an incremental backup from: - -- In GitLab 14.9 and 14.10, use the `BACKUP=<timestamp_of_backup>` option to choose the backup to use. The chosen previous backup is overwritten. -- In GitLab 15.0 and later, use the `PREVIOUS_BACKUP=<timestamp_of_backup>` option to choose the backup to use. By default, a backup file is created - as documented in the [Backup timestamp](#backup-timestamp) section. You can override the `[TIMESTAMP]` portion of the filename by setting the - [`BACKUP` environment variable](#backup-filename). - -To create an incremental backup, run: - -```shell -sudo gitlab-backup create INCREMENTAL=yes PREVIOUS_BACKUP=<timestamp_of_backup> -``` - -Incremental backups can also be created from [an untarred backup](#skipping-tar-creation) by using `SKIP=tar`: - -```shell -sudo gitlab-backup create INCREMENTAL=yes SKIP=tar -``` - -#### Back up specific repository storages - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86896) in GitLab 15.0. - -When using [multiple repository storages](../administration/repository_storage_paths.md), -repositories from specific repository storages can be backed up separately -using the `REPOSITORIES_STORAGES` option. The option accepts a comma-separated list of -storage names. - -For example, for Omnibus GitLab installations: - -```shell -sudo gitlab-backup create REPOSITORIES_STORAGES=storage1,storage2 -``` - -For example, for installations from source: - -```shell -sudo -u git -H bundle exec rake gitlab:backup:create REPOSITORIES_STORAGES=storage1,storage2 -``` - -#### Back up specific repositories - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88094) in GitLab 15.1. - -You can back up a specific repositories using the `REPOSITORIES_PATHS` option. -The option accepts a comma-separated list of project and group paths. If you -specify a group path, all repositories in all projects in the group and -descendent groups are included. - -For example, to back up all repositories for all projects in **Group A** (`group-a`), and the repository for **Project C** in **Group B** (`group-b/project-c`): - -- Omnibus GitLab installations: - - ```shell - sudo gitlab-backup create REPOSITORIES_PATHS=group-a,group-b/project-c - ``` - -- Installations from source: - - ```shell - sudo -u git -H bundle exec rake gitlab:backup:create REPOSITORIES_PATHS=group-a,group-b/project-c - ``` - -#### Uploading backups to a remote (cloud) storage - -NOTE: -It is not possible to [skip the tar creation](#skipping-tar-creation) when using object storage for backups. - -You can let the backup script upload (using the [Fog library](https://fog.io/)) -the `.tar` file it creates. In the following example, we use Amazon S3 for -storage, but Fog also lets you use [other storage providers](https://fog.io/storage/). -GitLab also [imports cloud drivers](https://gitlab.com/gitlab-org/gitlab/-/blob/da46c9655962df7d49caef0e2b9f6bbe88462a02/Gemfile#L113) -for AWS, Google, OpenStack Swift, Rackspace, and Aliyun. A local driver is -[also available](#uploading-to-locally-mounted-shares). - -[Read more about using object storage with GitLab](../administration/object_storage.md). - -##### Using Amazon S3 - -For Omnibus GitLab packages: - -1. Add the following to `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['backup_upload_connection'] = { - 'provider' => 'AWS', - 'region' => 'eu-west-1', - 'aws_access_key_id' => 'AKIAKIAKI', - 'aws_secret_access_key' => 'secret123' - # If using an IAM Profile, don't configure aws_access_key_id & aws_secret_access_key - # 'use_iam_profile' => true - } - gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' - ``` - -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect - -##### S3 Encrypted Buckets - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64765) in GitLab 14.3. - -AWS supports these [modes for server side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html): - -- Amazon S3-Managed Keys (SSE-S3) -- Customer Master Keys (CMKs) Stored in AWS Key Management Service (SSE-KMS) -- Customer-Provided Keys (SSE-C) - -Use your mode of choice with GitLab. Each mode has similar, but slightly -different, configuration methods. - -###### SSE-S3 - -To enable SSE-S3, in the backup storage options set the `server_side_encryption` -field to `AES256`. For example, in Omnibus GitLab: - -```ruby -gitlab_rails['backup_upload_storage_options'] = { - 'server_side_encryption' => 'AES256' -} -``` - -###### SSE-KMS - -To enable SSE-KMS, you'll need the [KMS key via its Amazon Resource Name (ARN) -in the `arn:aws:kms:region:acct-id:key/key-id` format](https://docs.aws.amazon.com/AmazonS3/latest/userguide/UsingKMSEncryption.html). Under the `backup_upload_storage_options` configuration setting, set: - -- `server_side_encryption` to `aws:kms`. -- `server_side_encryption_kms_key_id` to the ARN of the key. - -For example, in Omnibus GitLab: - -```ruby -gitlab_rails['backup_upload_storage_options'] = { - 'server_side_encryption' => 'aws:kms', - 'server_side_encryption_kms_key_id' => 'arn:aws:<YOUR KMS KEY ID>:' -} -``` - -###### SSE-C - -SSE-C requires you to set these encryption options: - -- `backup_encryption`: AES256. -- `backup_encryption_key`: Unencoded, 32-byte (256 bits) key. The upload fails if this isn't exactly 32 bytes. - -For example, in Omnibus GitLab: - -```ruby -gitlab_rails['backup_encryption'] = 'AES256' -gitlab_rails['backup_encryption_key'] = '<YOUR 32-BYTE KEY HERE>' -``` - -If the key contains binary characters and cannot be encoded in UTF-8, -instead, specify the key with the `GITLAB_BACKUP_ENCRYPTION_KEY` environment variable. -For example: +### `gitaly-backup` for repository backup and restore -```ruby -gitlab_rails['env'] = { 'GITLAB_BACKUP_ENCRYPTION_KEY' => "\xDE\xAD\xBE\xEF" * 8 } -``` +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333034) in GitLab 14.2. +> - [Deployed behind a feature flag](../user/feature_flags.md), enabled by default. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/333034) in GitLab 14.10. [Feature flag `gitaly_backup`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83254) removed. -##### Digital Ocean Spaces +The `gitaly-backup` binary is used by the backup Rake task to create and restore repository backups from Gitaly. +`gitaly-backup` replaces the previous backup method that directly calls RPCs on Gitaly from GitLab. -This example can be used for a bucket in Amsterdam (AMS3): +The backup Rake task must be able to find this executable. In most cases, you don't need to change +the path to the binary as it should work fine with the default path `/opt/gitlab/embedded/bin/gitaly-backup`. +If you have a specific reason to change the path, it can be configured in Omnibus GitLab packages: 1. Add the following to `/etc/gitlab/gitlab.rb`: ```ruby - gitlab_rails['backup_upload_connection'] = { - 'provider' => 'AWS', - 'region' => 'ams3', - 'aws_access_key_id' => 'AKIAKIAKI', - 'aws_secret_access_key' => 'secret123', - 'endpoint' => 'https://ams3.digitaloceanspaces.com' - } - gitlab_rails['backup_upload_remote_directory'] = 'my.s3.bucket' - ``` - -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect - -If you see a `400 Bad Request` error message when using Digital Ocean Spaces, -the cause may be the use of backup encryption. Because Digital Ocean Spaces -doesn't support encryption, remove or comment the line that contains -`gitlab_rails['backup_encryption']`. - -##### Other S3 Providers - -Not all S3 providers are fully compatible with the Fog library. For example, -if you see a `411 Length Required` error message after attempting to upload, -you may need to downgrade the `aws_signature_version` value from the default -value to `2`, [due to this issue](https://github.com/fog/fog-aws/issues/428). - -For installations from source: - -1. Edit `home/git/gitlab/config/gitlab.yml`: - - ```yaml - backup: - # snip - upload: - # Fog storage connection settings, see https://fog.io/storage/ . - connection: - provider: AWS - region: eu-west-1 - aws_access_key_id: AKIAKIAKI - aws_secret_access_key: 'secret123' - # If using an IAM Profile, leave aws_access_key_id & aws_secret_access_key empty - # ie. aws_access_key_id: '' - # use_iam_profile: 'true' - # The remote 'directory' to store your backups. For S3, this would be the bucket name. - remote_directory: 'my.s3.bucket' - # Specifies Amazon S3 storage class to use for backups, this is optional - # storage_class: 'STANDARD' - # - # Turns on AWS Server-Side Encryption with Amazon Customer-Provided Encryption Keys for backups, this is optional - # 'encryption' must be set in order for this to have any effect. - # 'encryption_key' should be set to the 256-bit encryption key for Amazon S3 to use to encrypt or decrypt. - # To avoid storing the key on disk, the key can also be specified via the `GITLAB_BACKUP_ENCRYPTION_KEY` your data. - # encryption: 'AES256' - # encryption_key: '<key>' - # - # - # Turns on AWS Server-Side Encryption with Amazon S3-Managed keys (optional) - # https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html - # For SSE-S3, set 'server_side_encryption' to 'AES256'. - # For SS3-KMS, set 'server_side_encryption' to 'aws:kms'. Set - # 'server_side_encryption_kms_key_id' to the ARN of customer master key. - # storage_options: - # server_side_encryption: 'aws:kms' - # server_side_encryption_kms_key_id: 'arn:aws:kms:YOUR-KEY-ID-HERE' - ``` - -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) - for the changes to take effect - -If you're uploading your backups to S3, you should create a new -IAM user with restricted access rights. To give the upload user access only for -uploading backups create the following IAM profile, replacing `my.s3.bucket` -with the name of your bucket: - -```json -{ - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "Stmt1412062044000", - "Effect": "Allow", - "Action": [ - "s3:AbortMultipartUpload", - "s3:GetBucketAcl", - "s3:GetBucketLocation", - "s3:GetObject", - "s3:GetObjectAcl", - "s3:ListBucketMultipartUploads", - "s3:PutObject", - "s3:PutObjectAcl" - ], - "Resource": [ - "arn:aws:s3:::my.s3.bucket/*" - ] - }, - { - "Sid": "Stmt1412062097000", - "Effect": "Allow", - "Action": [ - "s3:GetBucketLocation", - "s3:ListAllMyBuckets" - ], - "Resource": [ - "*" - ] - }, - { - "Sid": "Stmt1412062128000", - "Effect": "Allow", - "Action": [ - "s3:ListBucket" - ], - "Resource": [ - "arn:aws:s3:::my.s3.bucket" - ] - } - ] -} -``` - -##### Using Google Cloud Storage - -To use Google Cloud Storage to save backups, you must first create an -access key from the Google console: - -1. Go to the [Google storage settings page](https://console.cloud.google.com/storage/settings). -1. Select **Interoperability**, and then create an access key. -1. Make note of the **Access Key** and **Secret** and replace them in the - following configurations. -1. In the buckets advanced settings ensure the Access Control option - **Set object-level and bucket-level permissions** is selected. -1. Ensure you have already created a bucket. - -For Omnibus GitLab packages: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['backup_upload_connection'] = { - 'provider' => 'Google', - 'google_storage_access_key_id' => 'Access Key', - 'google_storage_secret_access_key' => 'Secret', - - ## If you have CNAME buckets (foo.example.com), you might run into SSL issues - ## when uploading backups ("hostname foo.example.com.storage.googleapis.com - ## does not match the server certificate"). In that case, uncomnent the following - ## setting. See: https://github.com/fog/fog/issues/2834 - #'path_style' => true - } - gitlab_rails['backup_upload_remote_directory'] = 'my.google.bucket' - ``` - -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect - -For installations from source: - -1. Edit `home/git/gitlab/config/gitlab.yml`: - - ```yaml - backup: - upload: - connection: - provider: 'Google' - google_storage_access_key_id: 'Access Key' - google_storage_secret_access_key: 'Secret' - remote_directory: 'my.google.bucket' - ``` - -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) - for the changes to take effect - -##### Using Azure Blob storage - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25877) in GitLab 13.4. - -For Omnibus GitLab packages: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['backup_upload_connection'] = { - 'provider' => 'AzureRM', - 'azure_storage_account_name' => '<AZURE STORAGE ACCOUNT NAME>', - 'azure_storage_access_key' => '<AZURE STORAGE ACCESS KEY>', - 'azure_storage_domain' => 'blob.core.windows.net', # Optional - } - gitlab_rails['backup_upload_remote_directory'] = '<AZURE BLOB CONTAINER>' - ``` - -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect - -For installations from source: - -1. Edit `home/git/gitlab/config/gitlab.yml`: - - ```yaml - backup: - upload: - connection: - provider: 'AzureRM' - azure_storage_account_name: '<AZURE STORAGE ACCOUNT NAME>' - azure_storage_access_key: '<AZURE STORAGE ACCESS KEY>' - remote_directory: '<AZURE BLOB CONTAINER>' - ``` - -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) - for the changes to take effect - -For more details, see the [table of Azure parameters](../administration/object_storage.md#azure-blob-storage). - -##### Specifying a custom directory for backups - -This option works only for remote storage. If you want to group your backups, -you can pass a `DIRECTORY` environment variable: - -```shell -sudo gitlab-backup create DIRECTORY=daily -sudo gitlab-backup create DIRECTORY=weekly -``` - -Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. - -#### Skip uploading backups to remote storage - -If you have configured GitLab to [upload backups in a remote storage](#uploading-backups-to-a-remote-cloud-storage), -you can use the `SKIP=remote` option to skip uploading your backups to the remote storage. - -For Omnibus GitLab packages: - -```shell -sudo gitlab-backup create SKIP=remote -``` - -For installations from source: - -```shell -sudo -u git -H bundle exec rake gitlab:backup:create SKIP=remote RAILS_ENV=production -``` - -#### Uploading to locally mounted shares - -You may also send backups to a mounted share (for example, `NFS`,`CIFS`, or -`SMB`) by using the Fog [`Local`](https://github.com/fog/fog-local#usage) -storage provider. The directory pointed to by the `local_root` key _must_ be -owned by the `git` user _when mounted_ (mounting with the `uid=` of the `git` -user for `CIFS` and `SMB`) or the user that you are executing the backup tasks -as (for Omnibus packages, this is the `git` user). - -The `backup_upload_remote_directory` _must_ be set in addition to the -`local_root` key. This is the sub directory inside the mounted directory that -backups are copied to, and is created if it does not exist. If the -directory that you want to copy the tarballs to is the root of your mounted -directory, use `.` instead. - -Because file system performance may affect overall GitLab performance, -[GitLab doesn't recommend using cloud-based file systems for storage](../administration/nfs.md#avoid-using-cloud-based-file-systems). - -For Omnibus GitLab packages: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['backup_upload_connection'] = { - :provider => 'Local', - :local_root => '/mnt/backups' - } - - # The directory inside the mounted folder to copy backups to - # Use '.' to store them in the root directory - gitlab_rails['backup_upload_remote_directory'] = 'gitlab_backups' - ``` - -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect. - -For installations from source: - -1. Edit `home/git/gitlab/config/gitlab.yml`: - - ```yaml - backup: - upload: - # Fog storage connection settings, see https://fog.io/storage/ . - connection: - provider: Local - local_root: '/mnt/backups' - # The directory inside the mounted folder to copy backups to - # Use '.' to store them in the root directory - remote_directory: 'gitlab_backups' - ``` - -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) - for the changes to take effect. - -#### Backup archive permissions - -The backup archives created by GitLab (`1393513186_2014_02_27_gitlab_backup.tar`) -have the owner/group `git`/`git` and 0600 permissions by default. This is -meant to avoid other system users reading GitLab data. If you need the backup -archives to have different permissions, you can use the `archive_permissions` -setting. - -For Omnibus GitLab packages: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['backup_archive_permissions'] = 0644 # Makes the backup archives world-readable + gitlab_rails['backup_gitaly_backup_path'] = '/path/to/gitaly-backup' ``` 1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. -For installations from source: - -1. Edit `/home/git/gitlab/config/gitlab.yml`: - - ```yaml - backup: - archive_permissions: 0644 # Makes the backup archives world-readable - ``` - -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) - for the changes to take effect. - -#### Configuring cron to make daily backups - -WARNING: -The following cron jobs do not [back up your GitLab configuration files](#storing-configuration-files) -or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). - -You can schedule a cron job that backs up your repositories and GitLab metadata. - -For Omnibus GitLab packages: - -1. Edit the crontab for the `root` user: - - ```shell - sudo su - - crontab -e - ``` - -1. There, add the following line to schedule the backup for everyday at 2 AM: - - ```plaintext - 0 2 * * * /opt/gitlab/bin/gitlab-backup create CRON=1 - ``` - - Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. - -For installations from source: - -1. Edit the crontab for the `git` user: - - ```shell - sudo -u git crontab -e - ``` - -1. Add the following lines at the bottom: - - ```plaintext - # Create a full backup of the GitLab repositories and SQL database every day at 2am - 0 2 * * * cd /home/git/gitlab && PATH=/usr/local/bin:/usr/bin:/bin bundle exec rake gitlab:backup:create RAILS_ENV=production CRON=1 - ``` - -The `CRON=1` environment setting directs the backup script to hide all progress -output if there aren't any errors. This is recommended to reduce cron spam. -When troubleshooting backup problems, however, replace `CRON=1` with `--trace` to log verbosely. - -### Limit backup lifetime for local files (prune old backups) - -WARNING: -The process described in this section don't work if you used a [custom filename](#backup-filename) -for your backups. - -To prevent regular backups from using all your disk space, you may want to set a limited lifetime -for backups. The next time the backup task runs, backups older than the `backup_keep_time` are -pruned. - -This configuration option manages only local files. GitLab doesn't prune old -files stored in a third-party [object storage](#uploading-backups-to-a-remote-cloud-storage) -because the user may not have permission to list and delete files. It's -recommended that you configure the appropriate retention policy for your object -storage (for example, [AWS S3](https://docs.aws.amazon.com/AmazonS3/latest/user-guide/create-lifecycle.html)). - -For Omnibus GitLab packages: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - ## Limit backup lifetime to 7 days - 604800 seconds - gitlab_rails['backup_keep_time'] = 604800 - ``` - -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect. +## Backup timestamp -For installations from source: +The backup archive is saved in `backup_path`, which is specified in the +`config/gitlab.yml` file. The filename is `[TIMESTAMP]_gitlab_backup.tar`, +where `TIMESTAMP` identifies the time at which each backup was created, plus +the GitLab version. The timestamp is needed if you need to restore GitLab and +multiple backups are available. -1. Edit `/home/git/gitlab/config/gitlab.yml`: +For example, if the backup name is `1493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar`, +the timestamp is `1493107454_2018_04_25_10.6.4-ce`. - ```yaml - backup: - ## Limit backup lifetime to 7 days - 604800 seconds - keep_time: 604800 - ``` +## Back up GitLab -1. [Restart GitLab](../administration/restart_gitlab.md#installations-from-source) - for the changes to take effect. +For detailed information on backing up GitLab, see [Backup GitLab](backup_gitlab.md). ## Restore GitLab -GitLab provides a command line interface to restore your entire installation, -and is flexible enough to fit your needs. - -The [restore prerequisites section](#restore-prerequisites) includes crucial -information. Be sure to read and test the complete restore process at least -once before attempting to perform it in a production environment. - -You can restore a backup only to _the exact same version and type (CE/EE)_ of -GitLab that you created it on (for example CE 9.1.0). - -If your backup is a different version than the current installation, you must -[downgrade your GitLab installation](../update/package/downgrade.md) -before restoring the backup. - -### Restore prerequisites - -You need to have a working GitLab installation before you can perform a -restore. This is because the system user performing the restore actions (`git`) -is usually not allowed to create or delete the SQL database needed to import -data into (`gitlabhq_production`). All existing data is either erased -(SQL) or moved to a separate directory (such as repositories and uploads). - -To restore a backup, you must restore `/etc/gitlab/gitlab-secrets.json` -(for Omnibus packages) or `/home/git/gitlab/.secret` (for installations from -source). This file contains the database encryption key, -[CI/CD variables](../ci/variables/index.md), and -variables used for [two-factor authentication](../user/profile/account/two_factor_authentication.md). -If you fail to restore this encryption key file along with the application data -backup, users with two-factor authentication enabled and GitLab Runner -loses access to your GitLab server. - -You may also want to restore your previous `/etc/gitlab/gitlab.rb` (for Omnibus packages) -or `/home/git/gitlab/config/gitlab.yml` (for installations from source) and -any TLS keys, certificates (`/etc/gitlab/ssl`, `/etc/gitlab/trusted-certs`), or -[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). - -Starting with GitLab 12.9, if an untarred backup (like the ones made with -`SKIP=tar`) is found, and no backup is chosen with `BACKUP=<timestamp>`, the -untarred backup is used. - -Depending on your case, you might want to run the restore command with one or -more of the following options: - -- `BACKUP=timestamp_of_backup`: Required if more than one backup exists. - Read what the [backup timestamp is about](#backup-timestamp). -- `force=yes`: Doesn't ask if the `authorized_keys` file should get regenerated, - and assumes 'yes' for warning about database tables being removed, - enabling the `Write to authorized_keys file` setting, and updating LDAP - providers. - -If you're restoring into directories that are mount points, you must ensure these directories are -empty before attempting a restore. Otherwise, GitLab attempts to move these directories before -restoring the new data, which causes an error. - -Read more about [configuring NFS mounts](../administration/nfs.md) - -### Restore for Omnibus GitLab installations - -This procedure assumes that: - -- You have installed the **exact same version and type (CE/EE)** of GitLab - Omnibus with which the backup was created. -- You have run `sudo gitlab-ctl reconfigure` at least once. -- GitLab is running. If not, start it using `sudo gitlab-ctl start`. - -First ensure your backup tar file is in the backup directory described in the -`gitlab.rb` configuration `gitlab_rails['backup_path']`. The default is -`/var/opt/gitlab/backups`. The backup file needs to be owned by the `git` user. - -```shell -sudo cp 11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar /var/opt/gitlab/backups/ -sudo chown git:git /var/opt/gitlab/backups/11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar -``` - -Stop the processes that are connected to the database. Leave the rest of GitLab -running: - -```shell -sudo gitlab-ctl stop puma -sudo gitlab-ctl stop sidekiq -# Verify -sudo gitlab-ctl status -``` - -Next, restore the backup, specifying the timestamp of the backup you wish to -restore: - -```shell -# This command will overwrite the contents of your GitLab database! -sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce -``` - -Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:restore` instead. -Some [known non-blocking error messages may appear](#restoring-database-backup-using-omnibus-packages-outputs-warnings). - -WARNING: -`gitlab-rake gitlab:backup:restore` doesn't set the correct file system -permissions on your Registry directory. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759). -In GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this -issue. - -If there's a GitLab version mismatch between your backup tar file and the -installed version of GitLab, the restore command aborts with an error -message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/), -and then try again. - -WARNING: -The restore command requires [additional parameters](#back-up-and-restore-for-installations-using-pgbouncer) when -your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster. - -Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary, -[as previously mentioned](#restore-prerequisites). - -Reconfigure, restart and [check](../administration/raketasks/maintenance.md#check-gitlab-configuration) GitLab: - -```shell -sudo gitlab-ctl reconfigure -sudo gitlab-ctl restart -sudo gitlab-rake gitlab:check SANITIZE=true -``` - -In GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets) -especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is -the target for the restore. - -```shell -sudo gitlab-rake gitlab:doctor:secrets -``` - -For added assurance, you can perform [an integrity check on the uploaded files](../administration/raketasks/check.md#uploaded-files-integrity): - -```shell -sudo gitlab-rake gitlab:artifacts:check -sudo gitlab-rake gitlab:lfs:check -sudo gitlab-rake gitlab:uploads:check -``` - -### Restore for Docker image and GitLab Helm chart installations - -For GitLab installations using the Docker image or the GitLab Helm chart on a -Kubernetes cluster, the restore task expects the restore directories to be -empty. However, with Docker and Kubernetes volume mounts, some system level -directories may be created at the volume roots, such as the `lost+found` -directory found in Linux operating systems. These directories are usually owned -by `root`, which can cause access permission errors since the restore Rake task -runs as the `git` user. To restore a GitLab installation, users have to confirm -the restore target directories are empty. - -For both these installation types, the backup tarball has to be available in -the backup location (default location is `/var/opt/gitlab/backups`). - -For Docker installations, the restore task can be run from host: - -```shell -# Stop the processes that are connected to the database -docker exec -it <name of container> gitlab-ctl stop puma -docker exec -it <name of container> gitlab-ctl stop sidekiq - -# Verify that the processes are all down before continuing -docker exec -it <name of container> gitlab-ctl status - -# Run the restore. NOTE: "_gitlab_backup.tar" is omitted from the name -docker exec -it <name of container> gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce - -# Restart the GitLab container -docker restart <name of container> - -# Check GitLab -docker exec -it <name of container> gitlab-rake gitlab:check SANITIZE=true -``` - -Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. - -WARNING: -`gitlab-rake gitlab:backup:restore` doesn't set the correct file system -permissions on your Registry directory. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759). -In GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this -issue. - -The GitLab Helm chart uses a different process, documented in -[restoring a GitLab Helm chart installation](https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/backup-restore/restore.md). - -### Restore for installation from source - -First, ensure your backup tar file is in the backup directory described in the -`gitlab.yml` configuration: - -```yaml -## Backup settings -backup: - path: "tmp/backups" # Relative paths are relative to Rails.root (default: tmp/backups/) -``` - -The default is `/home/git/gitlab/tmp/backups`, and it needs to be owned by the `git` user. Now, you can begin the backup procedure: - -```shell -# Stop processes that are connected to the database -sudo service gitlab stop - -sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production -``` - -Example output: - -```plaintext -Unpacking backup... [DONE] -Restoring database tables: --- create_table("events", {:force=>true}) - -> 0.2231s -[...] -- Loading fixture events...[DONE] -- Loading fixture issues...[DONE] -- Loading fixture keys...[SKIPPING] -- Loading fixture merge_requests...[DONE] -- Loading fixture milestones...[DONE] -- Loading fixture namespaces...[DONE] -- Loading fixture notes...[DONE] -- Loading fixture projects...[DONE] -- Loading fixture protected_branches...[SKIPPING] -- Loading fixture schema_migrations...[DONE] -- Loading fixture services...[SKIPPING] -- Loading fixture snippets...[SKIPPING] -- Loading fixture taggings...[SKIPPING] -- Loading fixture tags...[SKIPPING] -- Loading fixture users...[DONE] -- Loading fixture users_projects...[DONE] -- Loading fixture web_hooks...[SKIPPING] -- Loading fixture wikis...[SKIPPING] -Restoring repositories: -- Restoring repository abcd... [DONE] -- Object pool 1 ... -Deleting tmp directories...[DONE] -``` - -Next, restore `/home/git/gitlab/.secret` if necessary, [as previously mentioned](#restore-prerequisites). - -Restart GitLab: - -```shell -sudo service gitlab restart -``` - -### Restoring only one or a few projects or groups from a backup - -Although the Rake task used to restore a GitLab instance doesn't support -restoring a single project or group, you can use a workaround by restoring -your backup to a separate, temporary GitLab instance, and then export your -project or group from there: - -1. [Install a new GitLab](../install/index.md) instance at the same version as - the backed-up instance from which you want to restore. -1. [Restore the backup](#restore-gitlab) into this new instance, then - export your [project](../user/project/settings/import_export.md) - or [group](../user/group/settings/import_export.md). Be sure to read the - **Important Notes** on either export feature's documentation to understand - what is and isn't exported. -1. After the export is complete, go to the old instance and then import it. -1. After importing the projects or groups that you wanted is complete, you may - delete the new, temporary GitLab instance. - -A feature request to provide direct restore of individual projects or groups -is being discussed in [issue #17517](https://gitlab.com/gitlab-org/gitlab/-/issues/17517). - -### Restore options - -The command line tool GitLab provides to restore from backup can accept more -options. - -#### Excluding tasks on restore - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19347) in GitLab 14.10. - -You can exclude specific tasks on restore by adding the environment variable `SKIP`, whose values are a comma-separated list of the following options: - -- `db` (database) -- `uploads` (attachments) -- `builds` (CI job output logs) -- `artifacts` (CI job artifacts) -- `lfs` (LFS objects) -- `terraform_state` (Terraform states) -- `registry` (Container Registry images) -- `pages` (Pages content) -- `repositories` (Git repositories data) -- `packages` (Packages) - -For Omnibus GitLab packages: - -```shell -sudo gitlab-backup restore BACKUP=timestamp_of_backup SKIP=db,uploads -``` - -For installations from source: - -```shell -sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=timestamp_of_backup SKIP=db,uploads RAILS_ENV=production -``` - -#### Restore specific repository storages - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86896) in GitLab 15.0. - -When using [multiple repository storages](../administration/repository_storage_paths.md), -repositories from specific repository storages can be restored separately -using the `REPOSITORIES_STORAGES` option. The option accepts a comma-separated list of -storage names. - -For example, for Omnibus GitLab installations: - -```shell -sudo gitlab-backup restore BACKUP=timestamp_of_backup REPOSITORIES_STORAGES=storage1,storage2 -``` - -For example, for installations from source: - -```shell -sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=timestamp_of_backup REPOSITORIES_STORAGES=storage1,storage2 -``` - -#### Restore specific repositories - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88094) in GitLab 15.1. - -You can restore specific repositories using the `REPOSITORIES_PATHS` option. -The option accepts a comma-separated list of project and group paths. If you -specify a group path, all repositories in all projects in the group and -descendent groups are included. The project and group repositories must exist -within the specified backup. - -For example, to restore all repositories for all projects in **Group A** (`group-a`), and the repository for **Project C** in **Group B** (`group-b/project-c`): - -- Omnibus GitLab installations: - - ```shell - sudo gitlab-backup restore BACKUP=timestamp_of_backup REPOSITORIES_PATHS=group-a,group-b/project-c - ``` - -- Installations from source: - - ```shell - sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=timestamp_of_backup REPOSITORIES_PATHS=group-a,group-b/project-c - ``` +For detailed information on restoring GitLab, see [Restore GitLab](restore_gitlab.md). ## Alternative backup strategies @@ -1341,7 +106,7 @@ practical use. ### Back up repository data separately -First, ensure you back up existing GitLab data while [skipping repositories](#excluding-specific-directories-from-the-backup): +First, ensure you back up existing GitLab data while [skipping repositories](backup_gitlab.md#excluding-specific-directories-from-the-backup): ```shell # for Omnibus GitLab package installations @@ -1710,7 +475,7 @@ For more information, see: ### When the secrets file is lost -If you didn't [back up the secrets file](#storing-configuration-files), you +If you didn't [back up the secrets file](backup_gitlab.md#storing-configuration-files), you must complete several steps to get GitLab working properly again. The secrets file is responsible for storing the encryption key for the columns @@ -1754,7 +519,7 @@ Be sure to create a full database backup before attempting any changes. #### Disable user two-factor authentication (2FA) Users with 2FA enabled can't sign in to GitLab. In that case, you must -[disable 2FA for everyone](../security/two_factor_authentication.md#disable-2fa-for-everyone), +[disable 2FA for everyone](../security/two_factor_authentication.md#for-all-users), after which users must reactivate 2FA. #### Reset CI/CD variables @@ -1989,24 +754,222 @@ If this happens, examine the following: - If NFS is being used, check if the mount option `timeout` is set. The default is `600`, and changing this to smaller values results in this error. -### `gitaly-backup` for repository backup and restore +### Backup fails with `File name too long` error -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/333034) in GitLab 14.2. -> - [Deployed behind a feature flag](../user/feature_flags.md), enabled by default. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/333034) in GitLab 14.10. [Feature flag `gitaly_backup`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83254) removed. +During backup, you can get the `File name too long` error ([issue #354984](https://gitlab.com/gitlab-org/gitlab/-/issues/354984)). For example: -The `gitaly-backup` binary is used by the backup Rake task to create and restore repository backups from Gitaly. -`gitaly-backup` replaces the previous backup method that directly calls RPCs on Gitaly from GitLab. +```plaintext +Problem: <class 'OSError: [Errno 36] File name too long: +``` -The backup Rake task must be able to find this executable. In most cases, you don't need to change -the path to the binary as it should work fine with the default path `/opt/gitlab/embedded/bin/gitaly-backup`. -If you have a specific reason to change the path, it can be configured in Omnibus GitLab packages: +This problem stops the backup script from completing. To fix this problem, you must truncate the filenames causing the problem. A maximum of 246 characters, including the file extension, is permitted. -1. Add the following to `/etc/gitlab/gitlab.rb`: +WARNING: +The steps in this section can potentially lead to **data loss**. All steps must be followed strictly in the order given. - ```ruby - gitlab_rails['backup_gitaly_backup_path'] = '/path/to/gitaly-backup' +Truncating filenames to resolve the error involves: + +- Cleaning up remote uploaded files that aren't tracked in the database. +- Truncating the filenames in the database. +- Rerunning the backup task. + +#### Clean up remote uploaded files + +A [known issue](https://gitlab.com/gitlab-org/gitlab-ce/issues/45425) caused object store uploads to remain after a parent resource was deleted. This issue was [resolved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/18698). + +To fix these files, you must clean up all remote uploaded files that are in the storage but not tracked in the `uploads` database table. + +1. List all the object store upload files that can be moved to a lost and found directory if they don't exist in the GitLab database: + + ```shell + bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production ``` -1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) - for the changes to take effect +1. If you are sure you want to delete these files and remove all non-referenced uploaded files, run: + + WARNING: + The following action is **irreversible**. + + ```shell + bundle exec rake gitlab:cleanup:remote_upload_files RAILS_ENV=production DRY_RUN=false + ``` + +#### Truncate the filenames referenced by the database + +You must truncate the files referenced by the database that are causing the problem. The filenames referenced by the database are stored: + +- In the `uploads` table. +- In the references found. Any reference found from other database tables and columns. +- On the filesystem. + +Truncate the filenames in the `uploads` table: + +1. Enter the database console: + + For Omnibus GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For Omnibus GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + + For installations from source, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + + For installations from source, GitLab 14.1 and earlier: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production + ``` + +1. Search the `uploads` table for filenames longer than 246 characters: + + The following query selects the `uploads` records with filenames longer than 246 characters in batches of 0 to 10000. This improves the performance on large GitLab instances with tables having thousand of records. + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, id, path + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + SELECT + u.id, + u.path, + -- Current filename + (regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] AS current_filename, + -- New filename + CONCAT( + LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) AS new_filename, + -- New path + CONCAT( + COALESCE((regexp_match(u.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) AS new_path + FROM uploads_with_long_filenames AS u + WHERE u.row_id > 0 AND u.row_id <= 10000; + ``` + + Output example: + + ```postgresql + -[ RECORD 1 ]----+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- + id | 34 + path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt + current_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisit.txt + new_filename | loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt + new_path | public/@hashed/loremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelitsedvulputatemisitloremipsumdolorsitametconsecteturadipiscingelitseddoeiusmodtemporincididuntutlaboreetdoloremagnaaliquaauctorelits.txt + ``` + + Where: + + - `current_filename`: a filename that is currently more than 246 characters long. + - `new_filename`: a filename that has been truncated to 246 characters maximum. + - `new_path`: new path considering the new_filename (truncated). + + Once you validate the batch results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +1. Rename the files found in the `uploads` table from long filenames to new truncated filenames. The following query rolls back the update so you can check the results safely within a transaction wrapper: + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + BEGIN; + WITH updated_uploads AS ( + UPDATE uploads + SET + path = + CONCAT( + COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) + FROM + uploads_with_long_filenames AS updatable_uploads + WHERE + uploads.id = updatable_uploads.id + AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000 + RETURNING uploads.* + ) + SELECT id, path FROM updated_uploads; + ROLLBACK; + ``` + + Once you validate the batch update results, you must change the batch size (`row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +1. Validate that the new filenames from the previous query are the expected ones. If you are sure you want to truncate the records found in the previous step to 246 characters, run the following: + + WARNING: + The following action is **irreversible**. + + ```sql + CREATE TEMP TABLE uploads_with_long_filenames AS + SELECT ROW_NUMBER() OVER(ORDER BY id) row_id, path, id + FROM uploads AS u + WHERE LENGTH((regexp_match(u.path, '[^\\/:*?"<>|\r\n]+$'))[1]) > 246; + + CREATE INDEX ON uploads_with_long_filenames(row_id); + + UPDATE uploads + SET + path = + CONCAT( + COALESCE((regexp_match(updatable_uploads.path, '(.*\/).*'))[1], ''), + CONCAT( + LEFT(SPLIT_PART((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1], '.', 1), 242), + COALESCE(SUBSTRING((regexp_match(updatable_uploads.path, '[^\\/:*?"<>|\r\n]+$'))[1] FROM '\.(?:.(?!\.))+$')) + ) + ) + FROM + uploads_with_long_filenames AS updatable_uploads + WHERE + uploads.id = updatable_uploads.id + AND updatable_uploads.row_id > 0 AND updatable_uploads.row_id <= 10000; + ``` + + Once you finish the batch update, you must change the batch size (`updatable_uploads.row_id`) using the following sequence of numbers (10000 to 20000). Repeat this process until you reach the last record in the `uploads` table. + +Truncate the filenames in the references found: + +1. Check if those records are referenced somewhere. One way to do this is to dump the database and search for the parent directory name and filename: + + 1. To dump your database, you can use the following command as an example: + + ```shell + pg_dump -h /var/opt/gitlab/postgresql/ -d gitlabhq_production > gitlab-dump.tmp + ``` + + 1. Then you can search for the references using the `grep` command. Combining the parent directory and the filename can be a good idea. For example: + + ```shell + grep public/alongfilenamehere.txt gitlab-dump.tmp + ``` + +1. Replace those long filenames using the new filenames obtained from querying the `uploads` table. + +Truncate the filenames on the filesystem. You must manually rename the files in your filesystem to the new filenames obtained from querying the `uploads` table. + +#### Re-run the backup task + +After following all the previous steps, re-run the backup task. diff --git a/doc/raketasks/features.md b/doc/raketasks/features.md deleted file mode 100644 index e2554c1c18d..00000000000 --- a/doc/raketasks/features.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'index.md' -remove_date: '2022-05-24' ---- - -This document was moved to [another location](index.md). - -<!-- This redirect file can be deleted after <2022-05-24>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index e3acefb5520..5c95fe2eca1 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Rake tasks are available to import bare repositories into a GitLab instance. When migrating from an existing GitLab instance, and to preserve ownership by users and their namespaces, -please use [our project-based import/export](../user/project/settings/import_export.md). +use [our project-based import/export](../user/project/settings/import_export.md). Note that: @@ -39,7 +39,7 @@ To import bare repositories into a GitLab instance: - Groups are created as needed, these could be nested folders. For example, if we copy the repositories to `/var/opt/gitlab/git-data/repository-import-2020-08-22`, - and repository `A` needs to be under the groups `G1` and `G2`, it must be created under those folders: + and repository `A` must be under the groups `G1` and `G2`, it must be created under those folders: `/var/opt/gitlab/git-data/repository-import-2020-08-22/G1/G2/A.git`. ```shell @@ -49,7 +49,7 @@ To import bare repositories into a GitLab instance: sudo chown -R git:git /var/opt/gitlab/git-data/repository-import-$(date "+%Y-%m-%d") ``` - `foo.git` needs to be owned by the `git` user and `git` users group. + `foo.git` must be owned by the `git` user and `git` users group. If you are using an installation from source, replace `/var/opt/gitlab/` with `/home/git`. @@ -61,7 +61,7 @@ To import bare repositories into a GitLab instance: sudo gitlab-rake gitlab:import:repos["/var/opt/gitlab/git-data/repository-import-$(date "+%Y-%m-%d")"] ``` - - Installation from source. Before running this command you need to change to the directory where + - Installation from source. Before running this command you must change to the directory where your GitLab installation is located: ```shell diff --git a/doc/raketasks/index.md b/doc/raketasks/index.md index 38b57fab8ca..c2580e26ff0 100644 --- a/doc/raketasks/index.md +++ b/doc/raketasks/index.md @@ -27,7 +27,6 @@ The following Rake tasks are available for use with GitLab: | [Clean up](cleanup.md) | Clean up unneeded items from GitLab instances. | | [Development](../development/rake_tasks.md) | Tasks for GitLab contributors. | | [Elasticsearch](../integration/advanced_search/elasticsearch.md#gitlab-advanced-search-rake-tasks) | Maintain Elasticsearch in a GitLab instance. | -| [Enable namespaces](features.md) | Enable usernames and namespaces for user projects. | | [General maintenance](../administration/raketasks/maintenance.md) | General maintenance and self-check tasks. | | [Geo maintenance](../administration/raketasks/geo.md) | [Geo](../administration/geo/index.md)-related maintenance. | | [GitHub import](../administration/raketasks/github_import.md) | Retrieve and import repositories from GitHub. | diff --git a/doc/raketasks/restore_gitlab.md b/doc/raketasks/restore_gitlab.md new file mode 100644 index 00000000000..9d9a4d7b8c8 --- /dev/null +++ b/doc/raketasks/restore_gitlab.md @@ -0,0 +1,367 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Restore GitLab + +GitLab provides a command line interface to restore your entire installation, +and is flexible enough to fit your needs. + +The [restore prerequisites section](#restore-prerequisites) includes crucial +information. Be sure to read and test the complete restore process at least +once before attempting to perform it in a production environment. + +You can restore a backup only to _the exact same version and type (CE/EE)_ of +GitLab that you created it on (for example CE 9.1.0). + +If your backup is a different version than the current installation, you must +[downgrade your GitLab installation](../update/package/downgrade.md) +before restoring the backup. + +## Restore prerequisites + +You need to have a working GitLab installation before you can perform a +restore. This is because the system user performing the restore actions (`git`) +is usually not allowed to create or delete the SQL database needed to import +data into (`gitlabhq_production`). All existing data is either erased +(SQL) or moved to a separate directory (such as repositories and uploads). + +To restore a backup, you must restore `/etc/gitlab/gitlab-secrets.json` +(for Omnibus packages) or `/home/git/gitlab/.secret` (for installations from +source). This file contains the database encryption key, +[CI/CD variables](../ci/variables/index.md), and +variables used for [two-factor authentication](../user/profile/account/two_factor_authentication.md). +If you fail to restore this encryption key file along with the application data +backup, users with two-factor authentication enabled and GitLab Runner +loses access to your GitLab server. + +You may also want to restore your previous `/etc/gitlab/gitlab.rb` (for Omnibus packages) +or `/home/git/gitlab/config/gitlab.yml` (for installations from source) and +any TLS keys, certificates (`/etc/gitlab/ssl`, `/etc/gitlab/trusted-certs`), or +[SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). + +Starting with GitLab 12.9, if an untarred backup (like the ones made with +`SKIP=tar`) is found, and no backup is chosen with `BACKUP=<timestamp>`, the +untarred backup is used. + +Depending on your case, you might want to run the restore command with one or +more of the following options: + +- `BACKUP=timestamp_of_backup`: Required if more than one backup exists. + Read what the [backup timestamp is about](backup_restore.md#backup-timestamp). +- `force=yes`: Doesn't ask if the `authorized_keys` file should get regenerated, + and assumes 'yes' for warning about database tables being removed, + enabling the `Write to authorized_keys file` setting, and updating LDAP + providers. + +If you're restoring into directories that are mount points, you must ensure these directories are +empty before attempting a restore. Otherwise, GitLab attempts to move these directories before +restoring the new data, which causes an error. + +Read more about [configuring NFS mounts](../administration/nfs.md) + +## Restore for Omnibus GitLab installations + +This procedure assumes that: + +- You have installed the **exact same version and type (CE/EE)** of GitLab + Omnibus with which the backup was created. +- You have run `sudo gitlab-ctl reconfigure` at least once. +- GitLab is running. If not, start it using `sudo gitlab-ctl start`. + +First ensure your backup tar file is in the backup directory described in the +`gitlab.rb` configuration `gitlab_rails['backup_path']`. The default is +`/var/opt/gitlab/backups`. The backup file needs to be owned by the `git` user. + +```shell +sudo cp 11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar /var/opt/gitlab/backups/ +sudo chown git:git /var/opt/gitlab/backups/11493107454_2018_04_25_10.6.4-ce_gitlab_backup.tar +``` + +Stop the processes that are connected to the database. Leave the rest of GitLab +running: + +```shell +sudo gitlab-ctl stop puma +sudo gitlab-ctl stop sidekiq +# Verify +sudo gitlab-ctl status +``` + +Next, restore the backup, specifying the timestamp of the backup you wish to +restore: + +```shell +# This command will overwrite the contents of your GitLab database! +sudo gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce +``` + +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:restore` instead. +Some [known non-blocking error messages may appear](backup_restore.md#restoring-database-backup-using-omnibus-packages-outputs-warnings). + +WARNING: +`gitlab-rake gitlab:backup:restore` doesn't set the correct file system +permissions on your Registry directory. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759). +In GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this +issue. + +If there's a GitLab version mismatch between your backup tar file and the +installed version of GitLab, the restore command aborts with an error +message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab/), +and then try again. + +WARNING: +The restore command requires [additional parameters](backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer) when +your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster. + +Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary, +[as previously mentioned](#restore-prerequisites). + +Reconfigure, restart and [check](../administration/raketasks/maintenance.md#check-gitlab-configuration) GitLab: + +```shell +sudo gitlab-ctl reconfigure +sudo gitlab-ctl restart +sudo gitlab-rake gitlab:check SANITIZE=true +``` + +In GitLab 13.1 and later, check [database values can be decrypted](../administration/raketasks/check.md#verify-database-values-can-be-decrypted-using-the-current-secrets) +especially if `/etc/gitlab/gitlab-secrets.json` was restored, or if a different server is +the target for the restore. + +```shell +sudo gitlab-rake gitlab:doctor:secrets +``` + +For added assurance, you can perform [an integrity check on the uploaded files](../administration/raketasks/check.md#uploaded-files-integrity): + +```shell +sudo gitlab-rake gitlab:artifacts:check +sudo gitlab-rake gitlab:lfs:check +sudo gitlab-rake gitlab:uploads:check +``` + +## Restore for Docker image and GitLab Helm chart installations + +For GitLab installations using the Docker image or the GitLab Helm chart on a +Kubernetes cluster, the restore task expects the restore directories to be +empty. However, with Docker and Kubernetes volume mounts, some system level +directories may be created at the volume roots, such as the `lost+found` +directory found in Linux operating systems. These directories are usually owned +by `root`, which can cause access permission errors since the restore Rake task +runs as the `git` user. To restore a GitLab installation, users have to confirm +the restore target directories are empty. + +For both these installation types, the backup tarball has to be available in +the backup location (default location is `/var/opt/gitlab/backups`). + +For Docker installations, the restore task can be run from host: + +```shell +# Stop the processes that are connected to the database +docker exec -it <name of container> gitlab-ctl stop puma +docker exec -it <name of container> gitlab-ctl stop sidekiq + +# Verify that the processes are all down before continuing +docker exec -it <name of container> gitlab-ctl status + +# Run the restore. NOTE: "_gitlab_backup.tar" is omitted from the name +docker exec -it <name of container> gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce + +# Restart the GitLab container +docker restart <name of container> + +# Check GitLab +docker exec -it <name of container> gitlab-rake gitlab:check SANITIZE=true +``` + +Users of GitLab 12.1 and earlier should use the command `gitlab-rake gitlab:backup:create` instead. + +WARNING: +`gitlab-rake gitlab:backup:restore` doesn't set the correct file system +permissions on your Registry directory. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62759). +In GitLab 12.2 or later, you can use `gitlab-backup restore` to avoid this +issue. + +The GitLab Helm chart uses a different process, documented in +[restoring a GitLab Helm chart installation](https://gitlab.com/gitlab-org/charts/gitlab/blob/master/doc/backup-restore/restore.md). + +## Restore for installation from source + +First, ensure your backup tar file is in the backup directory described in the +`gitlab.yml` configuration: + +```yaml +## Backup settings +backup: + path: "tmp/backups" # Relative paths are relative to Rails.root (default: tmp/backups/) +``` + +The default is `/home/git/gitlab/tmp/backups`, and it needs to be owned by the `git` user. Now, you can begin the backup procedure: + +```shell +# Stop processes that are connected to the database +sudo service gitlab stop + +sudo -u git -H bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +Example output: + +```plaintext +Unpacking backup... [DONE] +Restoring database tables: +-- create_table("events", {:force=>true}) + -> 0.2231s +[...] +- Loading fixture events...[DONE] +- Loading fixture issues...[DONE] +- Loading fixture keys...[SKIPPING] +- Loading fixture merge_requests...[DONE] +- Loading fixture milestones...[DONE] +- Loading fixture namespaces...[DONE] +- Loading fixture notes...[DONE] +- Loading fixture projects...[DONE] +- Loading fixture protected_branches...[SKIPPING] +- Loading fixture schema_migrations...[DONE] +- Loading fixture services...[SKIPPING] +- Loading fixture snippets...[SKIPPING] +- Loading fixture taggings...[SKIPPING] +- Loading fixture tags...[SKIPPING] +- Loading fixture users...[DONE] +- Loading fixture users_projects...[DONE] +- Loading fixture web_hooks...[SKIPPING] +- Loading fixture wikis...[SKIPPING] +Restoring repositories: +- Restoring repository abcd... [DONE] +- Object pool 1 ... +Deleting tmp directories...[DONE] +``` + +Next, restore `/home/git/gitlab/.secret` if necessary, [as previously mentioned](#restore-prerequisites). + +Restart GitLab: + +```shell +sudo service gitlab restart +``` + +## Restoring only one or a few projects or groups from a backup + +Although the Rake task used to restore a GitLab instance doesn't support +restoring a single project or group, you can use a workaround by restoring +your backup to a separate, temporary GitLab instance, and then export your +project or group from there: + +1. [Install a new GitLab](../install/index.md) instance at the same version as + the backed-up instance from which you want to restore. +1. [Restore the backup](#restore-gitlab) into this new instance, then + export your [project](../user/project/settings/import_export.md) + or [group](../user/group/settings/import_export.md). Be sure to read the + **Important Notes** on either export feature's documentation to understand + what is and isn't exported. +1. After the export is complete, go to the old instance and then import it. +1. After importing the projects or groups that you wanted is complete, you may + delete the new, temporary GitLab instance. + +A feature request to provide direct restore of individual projects or groups +is being discussed in [issue #17517](https://gitlab.com/gitlab-org/gitlab/-/issues/17517). + +## Restore options + +The command line tool GitLab provides to restore from backup can accept more +options. + +### Disabling prompts during restore + +During a restore from backup, the restore script may ask for confirmation before +proceeding. If you wish to disable these prompts, you can set the `GITLAB_ASSUME_YES` +environment variable to `1`. + +For Omnibus GitLab packages: + +```shell +sudo GITLAB_ASSUME_YES=1 gitlab-backup restore +``` + +For installations from source: + +```shell +sudo -u git -H GITLAB_ASSUME_YES=1 bundle exec rake gitlab:backup:restore RAILS_ENV=production +``` + +### Excluding tasks on restore + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/19347) in GitLab 14.10. + +You can exclude specific tasks on restore by adding the environment variable `SKIP`, whose values are a comma-separated list of the following options: + +- `db` (database) +- `uploads` (attachments) +- `builds` (CI job output logs) +- `artifacts` (CI job artifacts) +- `lfs` (LFS objects) +- `terraform_state` (Terraform states) +- `registry` (Container Registry images) +- `pages` (Pages content) +- `repositories` (Git repositories data) +- `packages` (Packages) + +For Omnibus GitLab packages: + +```shell +sudo gitlab-backup restore BACKUP=timestamp_of_backup SKIP=db,uploads +``` + +For installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=timestamp_of_backup SKIP=db,uploads RAILS_ENV=production +``` + +### Restore specific repository storages + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86896) in GitLab 15.0. + +When using [multiple repository storages](../administration/repository_storage_paths.md), +repositories from specific repository storages can be restored separately +using the `REPOSITORIES_STORAGES` option. The option accepts a comma-separated list of +storage names. + +For example, for Omnibus GitLab installations: + +```shell +sudo gitlab-backup restore BACKUP=timestamp_of_backup REPOSITORIES_STORAGES=storage1,storage2 +``` + +For example, for installations from source: + +```shell +sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=timestamp_of_backup REPOSITORIES_STORAGES=storage1,storage2 +``` + +### Restore specific repositories + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/88094) in GitLab 15.1. + +You can restore specific repositories using the `REPOSITORIES_PATHS` option. +The option accepts a comma-separated list of project and group paths. If you +specify a group path, all repositories in all projects in the group and +descendent groups are included. The project and group repositories must exist +within the specified backup. + +For example, to restore all repositories for all projects in **Group A** (`group-a`), and the repository for **Project C** in **Group B** (`group-b/project-c`): + +- Omnibus GitLab installations: + + ```shell + sudo gitlab-backup restore BACKUP=timestamp_of_backup REPOSITORIES_PATHS=group-a,group-b/project-c + ``` + +- Installations from source: + + ```shell + sudo -u git -H bundle exec rake gitlab:backup:restore BACKUP=timestamp_of_backup REPOSITORIES_PATHS=group-a,group-b/project-c + ``` diff --git a/doc/raketasks/user_management.md b/doc/raketasks/user_management.md index b47afef7145..02c76af1e69 100644 --- a/doc/raketasks/user_management.md +++ b/doc/raketasks/user_management.md @@ -111,7 +111,7 @@ the leaked key without forcing all users to change their 2FA details. To rotate the two-factor authentication encryption key: -1. Look up the old key. This is in the `config/secrets.yml` file, but **make sure you're working +1. Look up the old key in the `config/secrets.yml` file, but **make sure you're working with the production section**. The line you're interested in looks like this: ```yaml diff --git a/doc/security/password_storage.md b/doc/security/password_storage.md index b4c2e27c952..d3db8cbe4f6 100644 --- a/doc/security/password_storage.md +++ b/doc/security/password_storage.md @@ -7,18 +7,32 @@ type: reference # Password storage **(FREE)** +> PBKDF2 and SHA512 [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/360658) in GitLab 15.2 [with flags](../administration/feature_flags.md) named `pbkdf2_password_encryption` and `pbkdf2_password_encryption_write`. Disabled by default. + GitLab stores user passwords in a hashed format to prevent passwords from being stored as plain text. GitLab uses the [Devise](https://github.com/heartcombo/devise) authentication library to hash user passwords. Created password hashes have these attributes: -- **Hashing**: The [`bcrypt`](https://en.wikipedia.org/wiki/Bcrypt) hashing - function is used to generate the hash of the provided password. This is a - strong, industry-standard cryptographic hashing function. +- **Hashing**: + - **BCrypt**: By default, the [`bcrypt`](https://en.wikipedia.org/wiki/Bcrypt) hashing + function is used to generate the hash of the provided password. This is a + strong, industry-standard cryptographic hashing function. + - **PBKDF2 and SHA512**: Starting in GitLab 15.2, PBKDF2 and SHA512 are supported + behind the following feature flags (disabled by default): + - `pbkdf2_password_encryption` - Enables reading and comparison of PBKDF2 + SHA512 + hashed passwords and supports fallback for BCrypt hashed passwords. + - `pbkdf2_password_encryption_write` - Enables new passwords to be saved + using PBKDF2 and SHA512, and existing BCrypt passwords to be migrated when users sign in. + + FLAG: + On self-managed GitLab, by default this feature is not available. To make it available, + ask an administrator to [enable the feature flags](../administration/feature_flags.md) named `pbkdf2_password_encryption` and `pbkdf2_password_encryption_write`. + - **Stretching**: Password hashes are [stretched](https://en.wikipedia.org/wiki/Key_stretching) to harden against brute-force attacks. By default, GitLab uses a stretching - factor of 10. + factor of 10 for BCrypt and 20,000 for PBKDF2 + SHA512. - **Salting**: A [cryptographic salt](https://en.wikipedia.org/wiki/Salt_(cryptography)) is added to each password to harden against pre-computed hash and dictionary attacks. To increase security, each salt is randomly generated for each diff --git a/doc/security/rate_limits.md b/doc/security/rate_limits.md index 695a0d52af6..e48a9999a06 100644 --- a/doc/security/rate_limits.md +++ b/doc/security/rate_limits.md @@ -43,6 +43,7 @@ You can set these rate limits in the Admin Area of your instance: - [Deprecated API rate limits](../user/admin_area/settings/deprecated_api_rate_limits.md) - [GitLab Pages rate limits](../administration/pages/index.md#rate-limits) - [Pipeline rate limits](../user/admin_area/settings/rate_limit_on_pipelines_creation.md) +- [Incident management rate limits](../user/admin_area/settings/incident_management_rate_limits.md) You can set these rate limits using the Rails console: diff --git a/doc/security/ssh_keys_restrictions.md b/doc/security/ssh_keys_restrictions.md index eb92694d236..eca52c41e4f 100644 --- a/doc/security/ssh_keys_restrictions.md +++ b/doc/security/ssh_keys_restrictions.md @@ -48,14 +48,10 @@ By default, the GitLab.com and self-managed settings for the - ECDSA_SK SSH keys are allowed (GitLab 14.8 and later). - ED25519_SK SSH keys are allowed (GitLab 14.8 and later). -### Block banned or compromised keys **(FREE)** +## Block banned or compromised keys **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24614) in GitLab 15.1 [with a flag](../administration/feature_flags.md) named `ssh_banned_key`. Enabled by default. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature per user, -ask an administrator to [disable the feature flag](../administration/feature_flags.md) named `ssh_banned_key`. -On GitLab.com, this feature is available. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24614) in GitLab 15.1 [with a flag](../administration/feature_flags.md) named `ssh_banned_key`. Enabled by default. +> - Generally available in GitLab 15.2. [Feature flag `ssh_banned_key`](https://gitlab.com/gitlab-org/gitlab/-/issues/363410) removed. When users attempt to [add a new SSH key](../user/ssh.md#add-an-ssh-key-to-your-gitlab-account) to GitLab accounts, the key is checked against a list of SSH keys which are known diff --git a/doc/security/two_factor_authentication.md b/doc/security/two_factor_authentication.md index ae13881fe6f..5907860f5cc 100644 --- a/doc/security/two_factor_authentication.md +++ b/doc/security/two_factor_authentication.md @@ -5,7 +5,7 @@ group: Authentication and Authorization info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Enforce two-factor authentication **(FREE SELF)** +# Enforce two-factor authentication **(FREE)** Two-factor authentication (2FA) provides an additional level of security to your users' GitLab account. When enabled, users are prompted for a code generated by an application in @@ -13,7 +13,7 @@ addition to supplying their username and password to sign in. Read more about [two-factor authentication (2FA)](../user/profile/account/two_factor_authentication.md) -## Enforce 2FA for all users +## Enforce 2FA for all users **(FREE SELF)** Users on GitLab can enable it without any administrator's intervention. If you want to enforce everyone to set up 2FA, you can choose from two different ways: @@ -33,7 +33,7 @@ To enable 2FA for all users: If you want 2FA enforcement to take effect during the next sign-in attempt, change the grace period to `0`. -## Disable 2FA enforcement through Rails console +### Disable 2FA enforcement through Rails console Using the [Rails console](../administration/operations/rails_console.md), enforcing 2FA for all user can be disabled. Connect to the Rails console and run: @@ -46,19 +46,22 @@ Gitlab::CurrentSettings.update!('require_two_factor_authentication': false) > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/24965) in GitLab 12.0, 2FA settings for a group are also applied to subgroups. -To enforce 2FA only for certain groups: +Prerequisites: + +- You must have the Maintainer or Owner role for the group. -1. Go to the group's **Settings > General** page. -1. Expand the **Permissions and group features** section. -1. Select the **Require all users in this group to set up two-factor authentication** option. +To enforce 2FA only for certain groups: -You can also specify a grace period in the **Time before enforced** option. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > General**. +1. Expand **Permissions and group features**. +1. Select **All users in this group must set up two-factor authentication**. +1. Select **Save changes**. -To change this setting, you must be an administrator or owner of the group. +You can also specify a grace period in the **Delay 2FA enforcement** option. If you want to enforce 2FA only for certain groups, you can enable it in the -group settings and specify a grace period as above. To change this setting you -must be administrator or owner of the group. +group settings and specify a grace period as above. The following are important notes about 2FA: @@ -80,15 +83,36 @@ The following are important notes about 2FA: 1. Uncheck the **Allow subgroups to set up their own two-factor authentication rule** field. This action causes all subgroups with 2FA requirements to stop requiring that from their members. +- Access tokens are not required to provide a second factor for authentication because they are API-based. + Tokens generated before 2FA is enforced remain valid. -## Disable 2FA for everyone +## Disable 2FA **(FREE SELF)** WARNING: -Disabling 2FA for everyone does not disable the [enforce 2FA for all users](#enforce-2fa-for-all-users) +Disabling 2FA for users does not disable the [enforce 2FA for all users](#enforce-2fa-for-all-users) or [enforce 2FA for all users in a group](#enforce-2fa-for-all-users-in-a-group) settings. You must also disable any enforced 2FA settings so users aren't asked to set up 2FA again when they next sign in to GitLab. +WARNING: +This is a permanent and irreversible action. Users must reactivate 2FA to use it again. + +### For a single user + +To disable 2FA for non-administrator users, we recommend using the [API endpoint](../api/users.md#disable-two-factor-authentication) +instead of the Rails console. +Using the [Rails console](../administration/operations/rails_console.md), 2FA for a single user can be disabled. +Connect to the Rails console and run: + +```ruby +admin = User.find_by_username('<USERNAME>') +user_to_disable = User.find_by_username('<USERNAME>') + +TwoFactor::DestroyService.new(admin, user: user_to_disable).execute +``` + +### For all users + There may be some special situations where you want to disable 2FA for everyone even when forced 2FA is disabled. There is a Rake task for that: @@ -100,10 +124,6 @@ sudo gitlab-rake gitlab:two_factor:disable_for_all_users sudo -u git -H bundle exec rake gitlab:two_factor:disable_for_all_users RAILS_ENV=production ``` -WARNING: -This is a permanent and irreversible action. Users have to -reactivate 2FA from scratch if they want to use it again. - ## 2FA for Git over SSH operations **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270554) in GitLab 13.7. diff --git a/doc/ssh/index.md b/doc/ssh/index.md deleted file mode 100644 index 10184a63a7a..00000000000 --- a/doc/ssh/index.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../user/ssh.md' -remove_date: '2022-06-18' ---- - -This document was moved to [another location](../user/ssh.md). - -<!-- This redirect file can be deleted after <2022-06-18>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/subscriptions/bronze_starter.md b/doc/subscriptions/bronze_starter.md index 6d88b61dd05..f936e230a3d 100644 --- a/doc/subscriptions/bronze_starter.md +++ b/doc/subscriptions/bronze_starter.md @@ -63,7 +63,7 @@ the tiers are no longer mentioned in GitLab documentation: - [`audit_json.log`](../administration/logs.md#audit_jsonlog) (specific entries) - [`elasticsearch.log`](../administration/logs.md#elasticsearchlog) - Merge requests: - - [Full code quality reports in the code quality tab](../user/project/merge_requests/code_quality.md#code-quality-reports) + - [Full code quality reports in the code quality tab](../ci/testing/code_quality.md#code-quality-reports) - [Merge request approvals](../user/project/merge_requests/approvals/index.md) - [Multiple assignees](../user/project/merge_requests/index.md#assign-multiple-users) - [Approval Rule information for Reviewers](../user/project/merge_requests/reviews/index.md#approval-rule-information-for-reviewers) diff --git a/doc/subscriptions/gitlab_dedicated/index.md b/doc/subscriptions/gitlab_dedicated/index.md index 0b3b44d74e1..218f6b7f824 100644 --- a/doc/subscriptions/gitlab_dedicated/index.md +++ b/doc/subscriptions/gitlab_dedicated/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab Dedicated NOTE: -GitLab Dedicated is currently in limited availability. Please [contact us](#contact-us) if you are interested. +GitLab Dedicated is currently in limited availability. [Contact us](#contact-us) if you are interested. GitLab Dedicated is a fully isolated, single-tenant SaaS service that is: @@ -18,7 +18,7 @@ GitLab Dedicated enables you to offload the operational overhead of managing the ## Available features -- Authentication: Support for instance-level [SAML OmniAuth](../../integration/saml.md) functionality. GitLab Dedicated acts as the service provider, and you will need to provide the necessary [configuration](../../integration/saml.md#general-setup) in order for GitLab to communicate with your IdP. This will be provided during onboarding. SAML [request signing](../../integration/saml.md#request-signing-optional) is supported. +- Authentication: Support for instance-level [SAML OmniAuth](../../integration/saml.md) functionality. GitLab Dedicated acts as the service provider, and you must provide the necessary [configuration](../../integration/saml.md#general-setup) in order for GitLab to communicate with your IdP. This is provided during onboarding. SAML [request signing](../../integration/saml.md#request-signing-optional) is supported. - Networking: - Public connectivity - Optional. Private connectivity via [AWS PrivateLink](https://aws.amazon.com/privatelink/). @@ -42,7 +42,7 @@ Features that are not available but we plan to support in the future: - GitLab-managed runners - FortiAuthenticator/FortiToken 2FA - Reply-by email -- Service desk +- Service Desk Features that we do not plan to offer at all: diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index d592c87d2a8..c1a11ce7d12 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -117,36 +117,34 @@ GitLab has several features which can help you manage the number of users: users manually. - View a breakdown of users by role in the [Users statistics](../../user/admin_area/index.md#users-statistics) page. -## Cloud licensing +## Sync your subscription data with GitLab > Introduced in GitLab 14.1. -Cloud licensing manages licenses for self-managed GitLab subscription plans. Cloud licensing includes: +Prerequisites: -- Activation: Unlock plan features and activate your self-managed instance by using an activation code. -- License sync: Sync subscription data between your self-managed instance and GitLab. +- You must be running GitLab Enterprise Edition (EE). +- You must have GitLab 14.1 or later. +- Your instance must be connected to the internet, and not be in an offline environment. -### How cloud licensing works +To sync subscription data between your self-managed instance and GitLab, you must [activate your instance](../../user/admin_area/license.md) with an +activation code. -#### Add your license +After you activate your instance, the following processes are automated: -1. When you purchase a GitLab self-managed plan, an activation code is generated. - This activation code is sent to the email address associated with the Customers Portal account. -1. In GitLab, on the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Subscription** and paste the activation code in the text field. -1. Select **Add license**. +- [Quarterly subscription reconciliation](../quarterly_reconciliation.md). +- Subscription renewals. +- Subscription updates, such as adding more seats or upgrading a GitLab tier. -The page displays the details of the subscription. +At approximately 03:00 UTC, a daily sync job sends subscription data to the Customers Portal. For this reason, updates and renewals may not +apply immediately. -#### License sync +The data is sent securely through an encrypted HTTPS connection to `customers.gitlab.com` on port `443`. +If the job fails, it retries up to 12 times over approximately 17 hours. -Once a day, a job sends license data to the Customers Portal. This information automates activation, -provisioning, co-terms, and renewals. The data is sent securely through an encrypted HTTPS connection -to `customers.gitlab.com` on port `443`. +### Subscription data that GitLab receives -This sync job runs daily around 3AM UTC. If the job fails, it is retried up to 12 times over approximately 17 hours. - -The daily job provides **only** the following information to the Customers Portal: +The daily sync job sends **only** the following information to the Customers Portal: - Date - Timestamp @@ -159,9 +157,8 @@ The daily job provides **only** the following information to the Customers Porta - GitLab version - Hostname - Instance ID -- MD5 hash of license -Example of a cloud licensing sync request: +Example of a license sync request: ```json { @@ -208,12 +205,16 @@ Example of a cloud licensing sync request: "max_historical_user_count": 75, "billable_users_count": 75, "hostname": "gitlab.example.com", - "instance_id": "9367590b-82ad-48cb-9da7-938134c29088", - "license_md5": "002f02470fe45ef6a333a4282aca6222" + "instance_id": "9367590b-82ad-48cb-9da7-938134c29088" } ``` -#### Sync subscription details +### Troubleshoot automatic subscription sync + +If the sync job is not working, ensure you allow network traffic from your GitLab instance +to IP address `104.18.26.123:443` (`customers.gitlab.com`). + +## Manually sync your subscription details You can manually sync your subscription details at any time. @@ -223,11 +224,6 @@ You can manually sync your subscription details at any time. A job is queued. When the job finishes, the subscription details are updated. -#### Troubleshooting cloud licensing sync - -If the sync job is not working, ensure you allow network traffic from your GitLab instance -to IP address `104.18.26.123:443` (`customers.gitlab.com`). - ## Obtain a subscription To subscribe to GitLab through a GitLab self-managed installation: @@ -273,7 +269,7 @@ If you are an administrator, you can export your license usage into a CSV: 1. On the left sidebar, select **Subscription**. 1. In the top right, select **Export license usage file**. -This file contains the information GitLab uses to manually process quarterly reconciliations or renewals. If your instance is firewalled or in an offline environment, you must provide GitLab with this information. +This file contains the information GitLab uses to manually process quarterly reconciliations or renewals. If your instance is firewalled or an offline environment, you must provide GitLab with this information. The **License Usage** CSV includes the following details: @@ -294,6 +290,8 @@ NOTES: ## Renew your subscription +You can renew your subscription starting from 15 days before your subscription expires. + To renew your subscription, [prepare for renewal by reviewing your account](#prepare-for-renewal-by-reviewing-your-account), then [renew your GitLab self-managed subscription](#renew-a-subscription). @@ -364,7 +362,8 @@ to your instance. ### Renew a subscription -Starting 30 days before a subscription expires, GitLab notifies administrators of the date of expiry with a banner in the GitLab user interface. +Starting 30 days before a subscription expires, a banner with the expiry date displays for administrators in the GitLab user interface. +You can renew your subscription starting from 15 days before your subscription expires. We recommend following these steps during renewal: @@ -375,7 +374,7 @@ The **Renew** button remains disabled (grayed-out) until 15 days before a subscr You can hover your mouse on the **Renew** button to see the date when it will become active. NOTE: - If you need to change your [GitLab tier](https://about.gitlab.com/pricing/), contact our sales team via [the sales contact form](https://about.gitlab.com/sales/) for assistance as this can't be done in the Customers Portal. + If you need to change your [GitLab tier](https://about.gitlab.com/pricing/), contact our sales team with [the sales contact form](https://about.gitlab.com/sales/) for assistance as this can't be done in the Customers Portal. 1. In the first box, enter the total number of user licenses you'll need for the upcoming year. Be sure this number is at least **equal to, or greater than** the number of billable users in the system at the time of performing the renewal. 1. Enter the number of [users over license](#users-over-license) in the second box for the user overage incurred in your previous subscription term. diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md deleted file mode 100644 index c30e2703a29..00000000000 --- a/doc/system_hooks/system_hooks.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../administration/system_hooks.md' -remove_date: '2022-06-18' ---- - -This document was moved to [another location](../administration/system_hooks.md). - -<!-- This redirect file can be deleted after <2022-06-18>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/tools/email.md b/doc/tools/email.md deleted file mode 100644 index 1d46a63bae4..00000000000 --- a/doc/tools/email.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../user/admin_area/email_from_gitlab.md' -remove_date: '2022-06-18' ---- - -This document was moved to [another location](../user/admin_area/email_from_gitlab.md). - -<!-- This redirect file can be deleted after <2022-06-18>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/topics/application_development_platform/index.md b/doc/topics/application_development_platform/index.md index 1560ceeed26..fac9f963a98 100644 --- a/doc/topics/application_development_platform/index.md +++ b/doc/topics/application_development_platform/index.md @@ -65,4 +65,3 @@ responsibility. The Application Development Platform integrates key performance into GitLab, automatically. The following features are included: - [Auto Monitoring](../autodevops/stages.md#auto-monitoring) -- [In-app Kubernetes Logs](../../user/project/clusters/kubernetes_pod_logs.md) diff --git a/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md b/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md index aceb1ed8910..b3ce2aa1683 100644 --- a/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md +++ b/doc/topics/autodevops/cloud_deployments/auto_devops_with_gke.md @@ -20,11 +20,12 @@ Ensure your own [runners are configured](../../../ci/runners/index.md) and To deploy a project to Google Kubernetes Engine, follow the steps below: 1. [Configure your Google account](#configure-your-google-account) -1. [Create a new project from a template](#create-a-new-project-from-a-template) -1. [Create a Kubernetes cluster from GitLab](#create-a-kubernetes-cluster-from-gitlab) +1. [Create a Kubernetes cluster and deploy the agent](#create-a-kubernetes-cluster) +1. [Create a new project from a template](#create-an-application-project-from-a-template) +1. [Configure the agent](#configure-the-agent) 1. [Install Ingress](#install-ingress) -1. [Configure your base domain](#configure-your-base-domain) -1. [Enable Auto DevOps](#enable-auto-devops-optional) +1. [Configure Auto DevOps](#configure-auto-devops) +1. [Enable Auto DevOps and run the pipeline](#enable-auto-devops-and-run-the-pipeline) 1. [Deploy the application](#deploy-the-application) ## Configure your Google account @@ -46,70 +47,45 @@ GCP accounts to get started with the GitLab integration with Google Kubernetes E [Follow this link](https://cloud.google.com/partners/partnercredit/?pcn_code=0014M00001h35gDQAQ#contact-form) and apply for credit. -## Create a new project from a template +## Create a Kubernetes cluster + +To create a new cluster on Google Kubernetes Engine (GKE), use Infrastructure as Code (IaC) approach +by following steps in [Create a Google GKE cluster](../../../user/infrastructure/clusters/connect/new_gke_cluster.md) guide. +The guide requires you to create a new project that uses [Terraform](https://www.terraform.io/) to create a GKE cluster and install a GitLab agent for Kubernetes. +This project is where configuration for GitLab agent resides. + +## Create an application project from a template Use a GitLab project template to get started. As the name suggests, those projects provide a bare-bones application built on some well-known frameworks. +WARNING: +Create the application project in the group hierarchy at the same level or below the project for cluster management. Otherwise, it fails to [authorize the agent](../../../user/clusters/agent/ci_cd_workflow.md#authorize-the-agent). + 1. On the top bar in GitLab, select the plus icon (**{plus-square}**), and select **New project/repository**. -1. Go to the **Create from template** tab, where you can choose a Ruby on - Rails, Spring, or NodeJS Express project. - For this tutorial, use the Ruby on Rails template. - - ![Select project template](img/guide_project_template_v12_3.png) - +1. Select **Create from template**. +1. Select the **Ruby on Rails** template. 1. Give your project a name, optionally a description, and make it public so that you can take advantage of the features available in the [GitLab Ultimate plan](https://about.gitlab.com/pricing/). - - ![Create project](img/guide_create_project_v12_3.png) - 1. Select **Create project**. -Now that you've created a project, create the Kubernetes cluster -to deploy this project to. - -## Create a Kubernetes cluster from GitLab - -1. On your project's landing page, select the button **Add Kubernetes cluster**. - - ![Project landing page](img/guide_project_landing_page_v12_10.png) +Now you have an application project you are going to deploy to the GKE cluster. -1. On the **Kubernetes clusters** page, select the **Create a new cluster** option from the **Actions** dropdown menu. +## Configure the agent -1. On the **Connect a Kubernetes cluster** page, select **Google GKE**. +Now we need to configure the GitLab agent for Kubernetes for us to be able to use it to deploy the application project. -1. Connect with your Google account, and select **Allow** to allow access to your - Google account. (This authorization request is only displayed the first time - you connect GitLab with your Google account.) +1. Go to the project [we created to manage the cluster](#create-a-kubernetes-cluster). +1. Navigate to the [agent configuration file](../../../user/clusters/agent/install/index.md#create-an-agent-configuration-file) (`.gitlab/agents/gke-agent/config.yaml`) and edit it. +1. Configure `ci_access:projects` attribute. Use application's project path as `id`: - After authorizing access, the **Connect a Kubernetes cluster** page - is displayed. - -1. In the **Enter your Kubernetes cluster certificate details** section, provide - details about your cluster: - - - **Kubernetes cluster name** - - **Environment scope** - Leave this field unchanged. - - **Google Cloud Platform project** - Select a project. When you - [configured your Google account](#configure-your-google-account), a project - should have already been created for you. - - **Zone** - The [region/zone](https://cloud.google.com/compute/docs/regions-zones/) to - create the cluster in. - - **Number of nodes** - - **Machine type** - For more information about - [machine types](https://cloud.google.com/compute/docs/machine-types), see Google's documentation. - - **Enable Cloud Run for Anthos** - Select this checkbox to use the - [Cloud Run](../../../user/project/clusters/add_gke_clusters.md#cloud-run-for-anthos), - Istio, and HTTP Load Balancing add-ons for this cluster. - - **GitLab-managed cluster** - Select this checkbox to - [allow GitLab to manage namespace and service accounts](../../../user/project/clusters/gitlab_managed_clusters.md) for this cluster. - -1. Select **Create Kubernetes cluster**. - -After a couple of minutes, the cluster is created. You can also see its -status on your [GCP dashboard](https://console.cloud.google.com/kubernetes). +```yaml +ci_access: + projects: + - id: path/to/application-project +``` ## Install Ingress @@ -134,9 +110,9 @@ or manually with Google Cloud Shell: kubectl get service ingress-nginx-ingress-controller -n gitlab-managed-apps ``` -## Configure your base domain +## Configure Auto DevOps -Follow these steps to configure the base domain where you access your apps. +Follow these steps to configure the base domain and other settings required for Auto DevOps. 1. A few minutes after you install NGINX, the load balancer obtains an IP address, and you can get the external IP address with the following command: @@ -149,30 +125,34 @@ Follow these steps to configure the base domain where you access your apps. Copy this IP address, as you need it in the next step. -1. Go back to the cluster page on GitLab, and go to the **Details** tab. - - Add your **Base domain**. For this example, use the domain `<IP address>.nip.io`. +1. Go back to the application project. +1. On the left sidebar, select **Settings > CI/CD** and expand **Variables**. + - Add a key called `KUBE_INGRESS_BASE_DOMAIN` with the application deployment domain as the value. For this example, use the domain `<IP address>.nip.io`. + - Add a key called `KUBE_NAMESPACE` with a value of the Kubernetes namespace for your deployments to target. You can use different namespaces per environment. Configure the environment, use the environment scope. + - Add a key called `KUBE_CONTEXT` with a value like `path/to/agent/project:gke-agent`. Select the environment scope of your choice. - Select **Save changes**. - ![Cluster Base Domain](img/guide_base_domain_v12_3.png) - -## Enable Auto DevOps (optional) +## Enable Auto DevOps and run the pipeline While Auto DevOps is enabled by default, Auto DevOps can be disabled at both the instance level (for self-managed instances) and the group level. Complete these steps to enable Auto DevOps if it's disabled: -1. Go to **Settings > CI/CD > Auto DevOps**, and select **Expand**. +1. On the top bar, select **Menu > Projects** and find the application project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **Auto DevOps**. 1. Select **Default to Auto DevOps pipeline** to display more options. 1. In **Deployment strategy**, select your desired [continuous deployment strategy](../requirements.md#auto-devops-deployment-strategy) to deploy the application to production after the pipeline successfully runs on the default branch. 1. Select **Save changes**. +1. Edit `.gitlab-ci.yml` file to include Auto DevOps template and commit the change to `master` branch: - ![Auto DevOps settings](img/guide_enable_autodevops_v12_3.png) - -After you save your changes, GitLab creates a new pipeline. To view it, go to -**{rocket}** **CI/CD > Pipelines**. + ```yaml + include: + - template: Auto-DevOps.gitlab-ci.yml + ``` -In the next section, we explain what each job does in the pipeline. +The commit should trigger a pipeline. In the next section, we explain what each job does in the pipeline. ## Deploy the application @@ -278,8 +258,6 @@ After submitting the merge request, GitLab runs your pipeline, and all the jobs in it, as [described previously](#deploy-the-application), in addition to a few more that run only on branches other than the default branch. -![Merge request](img/guide_merge_request_v12_3.png) - After a few minutes a test fails, which means a test was 'broken' by your change. Select the failed `test` job to see more information about it: @@ -311,8 +289,6 @@ see the test passing, but also the application deployed as a [review application](../stages.md#auto-review-apps). You can visit it by selecting the **View app** **{external-link}** button to see your changes deployed. -![Review app](img/guide_merge_request_review_app_v12_3.png) - After merging the merge request, GitLab runs the pipeline on the default branch, and then deploys the application to production. diff --git a/doc/topics/autodevops/cloud_deployments/img/guide_base_domain_v12_3.png b/doc/topics/autodevops/cloud_deployments/img/guide_base_domain_v12_3.png Binary files differdeleted file mode 100644 index 7d3b6a2f905..00000000000 --- a/doc/topics/autodevops/cloud_deployments/img/guide_base_domain_v12_3.png +++ /dev/null diff --git a/doc/topics/autodevops/cloud_deployments/img/guide_create_project_v12_3.png b/doc/topics/autodevops/cloud_deployments/img/guide_create_project_v12_3.png Binary files differdeleted file mode 100644 index a22730520ef..00000000000 --- a/doc/topics/autodevops/cloud_deployments/img/guide_create_project_v12_3.png +++ /dev/null diff --git a/doc/topics/autodevops/cloud_deployments/img/guide_enable_autodevops_v12_3.png b/doc/topics/autodevops/cloud_deployments/img/guide_enable_autodevops_v12_3.png Binary files differdeleted file mode 100644 index a3bcaeb99ae..00000000000 --- a/doc/topics/autodevops/cloud_deployments/img/guide_enable_autodevops_v12_3.png +++ /dev/null diff --git a/doc/topics/autodevops/cloud_deployments/img/guide_merge_request_review_app_v12_3.png b/doc/topics/autodevops/cloud_deployments/img/guide_merge_request_review_app_v12_3.png Binary files differdeleted file mode 100644 index e94654f4e50..00000000000 --- a/doc/topics/autodevops/cloud_deployments/img/guide_merge_request_review_app_v12_3.png +++ /dev/null diff --git a/doc/topics/autodevops/cloud_deployments/img/guide_merge_request_v12_3.png b/doc/topics/autodevops/cloud_deployments/img/guide_merge_request_v12_3.png Binary files differdeleted file mode 100644 index 5565be701cd..00000000000 --- a/doc/topics/autodevops/cloud_deployments/img/guide_merge_request_v12_3.png +++ /dev/null diff --git a/doc/topics/autodevops/cloud_deployments/img/guide_project_landing_page_v12_10.png b/doc/topics/autodevops/cloud_deployments/img/guide_project_landing_page_v12_10.png Binary files differdeleted file mode 100644 index 54e7141dad2..00000000000 --- a/doc/topics/autodevops/cloud_deployments/img/guide_project_landing_page_v12_10.png +++ /dev/null diff --git a/doc/topics/autodevops/cloud_deployments/img/guide_project_template_v12_3.png b/doc/topics/autodevops/cloud_deployments/img/guide_project_template_v12_3.png Binary files differdeleted file mode 100644 index 2b8d7224747..00000000000 --- a/doc/topics/autodevops/cloud_deployments/img/guide_project_template_v12_3.png +++ /dev/null diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index dfc2828e383..d8734ab5b13 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -243,29 +243,13 @@ See [Multiple Kubernetes clusters for Auto DevOps](multiple_clusters_auto_devops ## Customizing the Kubernetes namespace -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. +In GitLab 14.5 and earlier, you could use `environment:kubernetes:namespace` +to specify a namespace for the environment. +However, this feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8), +along with certificate-based integration. -For clusters not managed by GitLab, you can customize the namespace in -`.gitlab-ci.yml` by specifying -[`environment:kubernetes:namespace`](../../ci/environments/index.md#configure-kubernetes-deployments-deprecated). -For example, the following configuration overrides the namespace used for -`production` deployments: - -```yaml -include: - - template: Auto-DevOps.gitlab-ci.yml - -production: - environment: - kubernetes: - namespace: production -``` - -When deploying to a custom namespace with Auto DevOps, the service account -provided with the cluster needs at least the `edit` role within the namespace. - -- If the service account can create namespaces, then the namespace can be created on-demand. -- Otherwise, the namespace must exist prior to deployment. +You should now use the `KUBE_NAMESPACE` environment variable and +[limit the environments it is available for](../../ci/environments/index.md#scope-environments-with-specs). ## Using components of Auto DevOps @@ -315,7 +299,7 @@ You can configure many Auto DevOps jobs to run in an [offline environment](../.. build: image: "$REGISTRY_URL/docker/auto-build-image:v0.6.0" services: - - name: "$REGISTRY_URL/greg/docker/docker:20.10.6-dind" + - name: "$REGISTRY_URL/greg/docker/docker:20.10.16-dind" command: ['--tls=false', '--host=tcp://0.0.0.0:2375'] ``` diff --git a/doc/topics/autodevops/requirements.md b/doc/topics/autodevops/requirements.md index 02485ebafff..f3ea13ad1ce 100644 --- a/doc/topics/autodevops/requirements.md +++ b/doc/topics/autodevops/requirements.md @@ -27,7 +27,7 @@ To prepare the deployment: ## Auto DevOps deployment strategy -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/38542) in GitLab 11.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/38542) in GitLab 11.0. When using Auto DevOps to deploy your applications, choose the [continuous deployment strategy](../../ci/introduction/index.md) diff --git a/doc/topics/autodevops/stages.md b/doc/topics/autodevops/stages.md index a677787b980..8d35fd245d5 100644 --- a/doc/topics/autodevops/stages.md +++ b/doc/topics/autodevops/stages.md @@ -171,14 +171,14 @@ might want to use a [custom buildpack](customize.md#custom-buildpacks). ## Auto Code Quality -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) from GitLab Starter to GitLab Free in 13.2. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) from GitLab Starter to GitLab Free in 13.2. Auto Code Quality uses the [Code Quality image](https://gitlab.com/gitlab-org/ci-cd/codequality) to run static analysis and other code checks on the current code. After creating the report, it's uploaded as an artifact which you can later download and check out. The merge request widget also displays any -[differences between the source and target branches](../../user/project/merge_requests/code_quality.md). +[differences between the source and target branches](../../ci/testing/code_quality.md). ## Auto SAST @@ -336,7 +336,7 @@ You can disable DAST: > Introduced in GitLab 10.4. -Auto [Browser Performance Testing](../../user/project/merge_requests/browser_performance_testing.md) +Auto [Browser Performance Testing](../../ci/testing/browser_performance_testing.md) measures the browser performance of a web page with the [Sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/), creates a JSON report including the overall performance score for each page, and @@ -351,13 +351,13 @@ file named `.gitlab-urls.txt` in the root directory, one file per line. For exam ``` Any browser performance differences between the source and target branches are also -[shown in the merge request widget](../../user/project/merge_requests/browser_performance_testing.md). +[shown in the merge request widget](../../ci/testing/browser_performance_testing.md). ## Auto Load Performance Testing **(PREMIUM)** > Introduced in GitLab 13.2. -Auto [Load Performance Testing](../../user/project/merge_requests/load_performance_testing.md) +Auto [Load Performance Testing](../../ci/testing/load_performance_testing.md) measures the server performance of an application with the [k6 container](https://hub.docker.com/r/loadimpact/k6/), creates a JSON report including several key result metrics, and @@ -368,7 +368,7 @@ written that's tailored to your specific application. The test also needs to be configured so it can pick up the environment's dynamic URL via a CI/CD variable. Any load performance test result differences between the source and target branches are also -[shown in the merge request widget](../../user/project/merge_requests/load_performance_testing.md). +[shown in the merge request widget](../../user/project/merge_requests/widgets.md). ## Auto Deploy diff --git a/doc/topics/offline/quick_start_guide.md b/doc/topics/offline/quick_start_guide.md index 901a2dc750c..aa923eb6dc6 100644 --- a/doc/topics/offline/quick_start_guide.md +++ b/doc/topics/offline/quick_start_guide.md @@ -23,7 +23,7 @@ For a video walkthrough of this process, see [Offline GitLab Installation: Downl You should [manually download the GitLab package](../../update/package/index.md#upgrade-using-a-manually-downloaded-package) and relevant dependencies using a server of the same operating system type that has access to the Internet. -If your offline environment has no local network access, you must manually transport across the relevant package files through physical media, such as a USB drive or writable DVD. +If your offline environment has no local network access, you must manually transport across the relevant package files through physical media, such as a USB drive, or writable DVD. In Ubuntu, this can be performed on a server with Internet access using the following commands: @@ -70,7 +70,7 @@ sudo EXTERNAL_URL="http://my-host.internal" dpkg -i <gitlab_package_name>.deb ## Enabling SSL -Follow these steps to enable SSL for your fresh instance. Note that these steps reflect those for +Follow these steps to enable SSL for your fresh instance. These steps reflect those for [manually configuring SSL in Omnibus's NGINX configuration](https://docs.gitlab.com/omnibus/settings/nginx.html#manually-configuring-https): 1. Make the following changes to `/etc/gitlab/gitlab.rb`: @@ -100,7 +100,7 @@ Follow these steps to enable SSL for your fresh instance. Note that these steps ## Enabling the GitLab Container Registry -Follow these steps to enable the container registry. Note that these steps reflect those for +Follow these steps to enable the container registry. These steps reflect those for [configuring the container registry under an existing domain](../../administration/packages/container_registry.md#configure-container-registry-under-an-existing-gitlab-domain): 1. Make the following changes to `/etc/gitlab/gitlab.rb`: @@ -180,7 +180,7 @@ sudo docker run -d --restart always --name gitlab-runner -v /etc/gitlab-runner:/ ### Authenticating the registry against the host OS -As noted in [Docker's registry authentication documentation](https://docs.docker.com/registry/insecure/#docker-still-complains-about-the-certificate-when-using-authentication), +As noted in [Docker registry authentication documentation](https://docs.docker.com/registry/insecure/#docker-still-complains-about-the-certificate-when-using-authentication), certain versions of Docker require trusting the certificate chain at the OS level. In the case of Ubuntu, this involves using `update-ca-certificates`: diff --git a/doc/topics/use_gitlab.md b/doc/topics/use_gitlab.md deleted file mode 100644 index 6c6c5f71fc2..00000000000 --- a/doc/topics/use_gitlab.md +++ /dev/null @@ -1,11 +0,0 @@ ---- -redirect_to: '../user/index.md' -remove_date: '2022-06-17' ---- - -This document was moved to [another location](../user/index.md). - -<!-- This redirect file can be deleted after <2022-06-17>. --> -<!-- Redirects that point to other docs in the same project expire in three months. --> -<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/tutorials/make_your_first_git_commit.md b/doc/tutorials/make_your_first_git_commit.md index fd153b51a2c..a35137e158b 100644 --- a/doc/tutorials/make_your_first_git_commit.md +++ b/doc/tutorials/make_your_first_git_commit.md @@ -19,7 +19,7 @@ Before you begin: - [Install Git on your local machine](../topics/git/how_to_install_git/index.md). - Ensure you can sign in to an instance of GitLab. If your organization doesn't have GitLab, create an account on GitLab.com. -- [Create SSH keys and add them to GitLab](../ssh/index.md). SSH keys are how you +- [Create SSH keys and add them to GitLab](../user/ssh.md). SSH keys are how you securely communicate between your computer and GitLab. ## What is Git? diff --git a/doc/tutorials/move_personal_project_to_a_group.md b/doc/tutorials/move_personal_project_to_a_group.md index f8b91792780..fdda42be3fa 100644 --- a/doc/tutorials/move_personal_project_to_a_group.md +++ b/doc/tutorials/move_personal_project_to_a_group.md @@ -77,7 +77,7 @@ project. NOTE: For more information about these migration steps, -see [Transferring your project into another namespace](../user/project/settings/index.md#transferring-an-existing-project-into-another-namespace). +see [Transferring your project into another namespace](../user/project/settings/index.md#transfer-a-project-to-another-namespace). A migration might result in follow-up work to update the project path in your related resources and tools, such as websites and package managers. diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index d726f96f646..81b98b95068 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -45,6 +45,27 @@ sole discretion of GitLab Inc. <div class="announcement-milestone"> +## Announced in 15.2 + +<div class="deprecation removal-160 breaking-change"> + +### Remove `job_age` parameter from `POST /jobs/request` Runner endpoint + +Planned removal: GitLab <span class="removal-milestone">16.0</span> (2023-05-22) + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Review the details carefully before upgrading. + +The `job_age` parameter, returned from the `POST /jobs/request` API endpoint used in communication with GitLab Runner, was never used by any GitLab or Runner feature. This parameter will be removed in GitLab 16.0. + +This could be a breaking change for anyone that developed their own runner that relies on this parameter being returned by the endpoint. This is not a breaking change for anyone using an officially released version of GitLab Runner, including public shared runners on GitLab.com. + +</div> +</div> + +<div class="announcement-milestone"> + ## Announced in 15.1 <div class="deprecation removal-160 breaking-change"> @@ -892,11 +913,11 @@ If you have explicitly excluded retire.js using DS_EXCLUDED_ANALYZERS you will n </div> -<div class="deprecation removal-152 breaking-change"> +<div class="deprecation removal-154 breaking-change"> ### SAST analyzer consolidation and CI/CD template changes -Planned removal: GitLab <span class="removal-milestone">15.2</span> (2022-07-22) +Planned removal: GitLab <span class="removal-milestone">15.4</span> (2022-09-22) WARNING: This is a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). @@ -905,9 +926,9 @@ Review the details carefully before upgrading. GitLab SAST uses various [analyzers](https://docs.gitlab.com/ee/user/application_security/sast/analyzers/) to scan code for vulnerabilities. We are reducing the number of analyzers used in GitLab SAST as part of our long-term strategy to deliver a better and more consistent user experience. -Streamlining the set of analyzers will also enable faster [iteration](https://about.gitlab.com/handbook/values/#iteration), better [results](https://about.gitlab.com/handbook/values/#results), and greater [efficiency](https://about.gitlab.com/handbook/values/#results) (including a reduction in CI runner usage in most cases). +Streamlining the set of analyzers will also enable faster [iteration](https://about.gitlab.com/handbook/values/#iteration), better [results](https://about.gitlab.com/handbook/values/#results), and greater [efficiency](https://about.gitlab.com/handbook/values/#efficiency) (including a reduction in CI runner usage in most cases). -In GitLab 15.2, GitLab SAST will no longer use the following analyzers: +In GitLab 15.4, GitLab SAST will no longer use the following analyzers: - [ESLint](https://gitlab.com/gitlab-org/security-products/analyzers/eslint) (JavaScript, TypeScript, React) - [Gosec](https://gitlab.com/gitlab-org/security-products/analyzers/gosec) (Go) @@ -922,7 +943,14 @@ We will not delete container images previously published for these analyzers; an We will also remove Java from the scope of the [SpotBugs](https://gitlab.com/gitlab-org/security-products/analyzers/spotbugs) analyzer and replace it with the [Semgrep-based analyzer](https://gitlab.com/gitlab-org/security-products/analyzers/semgrep). This change will make it simpler to scan Java code; compilation will no longer be required. -This change will be reflected in the automatic language detection portion of the [GitLab-managed SAST CI/CD template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml). +This change will be reflected in the automatic language detection portion of the [GitLab-managed SAST CI/CD template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml). Note that the SpotBugs-based analyzer will continue to cover Groovy, Kotlin, and Scala. + +If you've already dismissed a vulnerability finding from one of the deprecated analyzers, the replacement attempts to respect your previous dismissal. The system behavior depends on: + +- whether you’ve excluded the Semgrep-based analyzer from running in the past. +- which analyzer first discovered the vulnerabilities shown in the project’s Vulnerability Report. + +See [Vulnerability translation documentation](https://docs.gitlab.com/ee/user/application_security/sast/analyzers.html#vulnerability-translation) for further details. If you applied customizations to any of the affected analyzers or if you currently disable the Semgrep analyzer in your pipelines, you must take action as detailed in the [deprecation issue for this change](https://gitlab.com/gitlab-org/gitlab/-/issues/352554#breaking-change). @@ -1453,12 +1481,12 @@ Tracing in GitLab is an integration with Jaeger, an open-source end-to-end distr <div class="deprecation removal-150"> -### `artifacts:report:cobertura` keyword +### `artifacts:reports:cobertura` keyword Planned removal: GitLab <span class="removal-milestone">15.0</span> (2022-05-22) Currently, test coverage visualizations in GitLab only support Cobertura reports. Starting 15.0, the -`artifacts:report:cobertura` keyword will be replaced by +`artifacts:reports:cobertura` keyword will be replaced by [`artifacts:reports:coverage_report`](https://gitlab.com/gitlab-org/gitlab/-/issues/344533). Cobertura will be the only supported report file in 15.0, but this is the first step towards GitLab supporting other report types. diff --git a/doc/update/index.md b/doc/update/index.md index dcdcf8f01ae..e9fa2321450 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -54,7 +54,7 @@ There are also instructions when you want to Editions. In the past we used separate documents for the upgrading instructions, but we -have since switched to using a single document. The old upgrading guidelines +have switched to using a single document. The old upgrading guidelines can still be found in the Git repository: - [Old upgrading guidelines for Community Edition](https://gitlab.com/gitlab-org/gitlab-foss/tree/11-8-stable/doc/update) @@ -63,8 +63,8 @@ can still be found in the Git repository: ### Installation using Docker GitLab provides official Docker images for both Community and Enterprise -editions. They are based on the Omnibus package and instructions on how to -update them are in [a separate document](https://docs.gitlab.com/omnibus/docker/README.html). +editions, and they are based on the Omnibus package. See how to +[install GitLab using Docker](../install/docker.md). ### Installation using Helm @@ -179,7 +179,7 @@ Expected batched background migration for the given configuration to be marked a If you get this error, [check the batched background migration options](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished) to complete the upgrade. -### What do I do if my background migrations are stuck? +### What do you do if your background migrations are stuck? WARNING: The following operations can disrupt your GitLab performance. They run a number of Sidekiq jobs that perform various database or file updates. @@ -331,11 +331,11 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:elastic:list_pending_migrations ``` -### What do I do if my Advanced Search migrations are stuck? +### What do you do if your Advanced Search migrations are stuck? See [how to retry a halted migration](../integration/advanced_search/elasticsearch.md#retry-a-halted-migration). -### What do I do for the error `Elasticsearch version not compatible` +### What do you do for the error `Elasticsearch version not compatible` Confirm that your version of Elasticsearch or OpenSearch is [compatible with your version of GitLab](../integration/advanced_search/elasticsearch.md#version-requirements). @@ -361,7 +361,7 @@ It's also important to ensure that any [background migrations have been fully co before upgrading to a new major version. If you have enabled the [Elasticsearch integration](../integration/advanced_search/elasticsearch.md) **(PREMIUM SELF)**, then -[ensure all Advanced Search migrations are completed](#checking-for-pending-advanced-search-migrations) in the last minor version within +[ensure all Advanced Search migrations are completed](#checking-for-pending-advanced-search-migrations) in the last minor version in your current version before proceeding with the major version upgrade. @@ -379,7 +379,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab accordingly, while also consulting the [version-specific upgrade instructions](#version-specific-upgrading-instructions): -`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> [`12.10.14`](#12100) -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](#13120) -> [`14.0.12`](#1400) -> [`14.3.6`](#1430) -> [`14.9.5`](#1490) -> [`14.10.Z`](#1410) -> [`15.0.Z`](#1500) -> [latest `15.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases) +`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> [`12.10.14`](#12100) -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](#13120) -> [`14.0.12`](#1400) -> [`14.3.6`](#1430) -> [`14.9.5`](#1490) -> [`14.10.Z`](#14100) -> [`15.0.Z`](#1500) -> [latest `15.Y.Z`](https://gitlab.com/gitlab-org/gitlab/-/releases) The following table, while not exhaustive, shows some examples of the supported upgrade paths. @@ -387,8 +387,8 @@ Additional steps between the mentioned versions are possible. We list the minima | Target version | Your version | Supported upgrade path | Note | | -------------- | ------------ | ---------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | -| `15.1.0` | `14.6.2` | `14.6.2` -> `14.9.5` -> `14.10.4` -> `15.0.2` -> `15.1.0` | Three intermediate versions are required: `14.9` and `14.10`, `15.0`, then `15.1.0`. | -| `15.0.0` | `14.6.2` | `14.6.2` -> `14.9.5` -> `14.10.4` -> `15.0.2` | Two intermediate versions are required: `14.9` and `14.10`, then `15.0.0`. | +| `15.1.0` | `14.6.2` | `14.6.2` -> `14.9.5` -> `14.10.5` -> `15.0.2` -> `15.1.0` | Three intermediate versions are required: `14.9` and `14.10`, `15.0`, then `15.1.0`. | +| `15.0.0` | `14.6.2` | `14.6.2` -> `14.9.5` -> `14.10.5` -> `15.0.2` | Two intermediate versions are required: `14.9` and `14.10`, then `15.0.0`. | | `14.6.2` | `13.10.2` | `13.10.2` -> `13.12.15` -> `14.0.12` -> `14.3.6` => `14.6.2` | Three intermediate versions are required: `13.12` and `14.0`, `14.3`, then `14.6.2`. | | `14.1.8` | `13.9.2` | `13.9.2` -> `13.12.15` -> `14.0.12` -> `14.1.8` | Two intermediate versions are required: `13.12` and `14.0`, then `14.1.8`. | | `13.12.15` | `12.9.2` | `12.9.2` -> `12.10.14` -> `13.0.14` -> `13.1.11` -> `13.8.8` -> `13.12.15` | Four intermediate versions are required: `12.10`, `13.0`, `13.1` and `13.8.8`, then `13.12.15`. | @@ -425,7 +425,7 @@ Edition, follow the guides below based on the installation method: ### Enterprise to Community Edition -If you need to downgrade your Enterprise Edition installation back to Community +To downgrade your Enterprise Edition installation back to Community Edition, you can follow [this guide](../downgrade_ee_to_ce/index.md) to make the process as smooth as possible. @@ -442,7 +442,7 @@ At the end of major and minor release posts, there are three sections to look fo These include: -- Steps you need to perform as part of an upgrade. +- Steps you must perform as part of an upgrade. For example [8.12](https://about.gitlab.com/releases/2016/09/22/gitlab-8-12-released/#upgrade-barometer) required the Elasticsearch index to be recreated. Any older version of GitLab upgrading to 8.12 or later would require this. - Changes to the versions of software we support such as @@ -458,11 +458,28 @@ NOTE: Specific information that follow related to Ruby and Git versions do not apply to [Omnibus installations](https://docs.gitlab.com/omnibus/) and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with appropriate Ruby and Git versions and are not using system binaries for Ruby and Git. There is no need to install Ruby or Git when utilizing these two approaches. +### 15.2.0 (unreleased) + +- GitLab installations that have multiple web nodes should be + [upgraded to 15.1](#1510) before upgrading to 15.2 (and later) due to a + configuration change in Rails that can result in inconsistent ETag key + generation. +- Some Sidekiq workers were renamed in this release. To avoid any disruption, [run the Rake tasks to migrate any pending jobs](../raketasks/sidekiq_job_migration.md#future-jobs) before starting the upgrade to GitLab 15.2.0. + ### 15.1.0 - If you run external PostgreSQL, particularly AWS RDS, [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue) to avoid the database crashing. +- In GitLab 15.1.0, we are switching Rails `ActiveSupport::Digest` to use SHA256 instead of MD5. + This affects ETag key generation for resources such as raw Snippet file + downloads. To ensure consistent ETag key generation across multiple + web nodes when upgrading, all servers must first be upgraded to 15.1.Z before + upgrading to 15.2.0 or later: + + 1. Ensure all GitLab web nodes are running GitLab 15.1.Z. + 1. [Enable the `active_support_hash_digest_sha256` feature flag](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) to switch `ActiveSupport::Digest` to use SHA256: + 1. Only then, continue to upgrade to later versions of GitLab. - Unauthenticated requests to the [`ciConfig` GraphQL field](../api/graphql/reference/index.md#queryciconfig) are no longer supported. Before you upgrade to GitLab 15.1, add an [access token](../api/index.md#authentication) to your requests. The user creating the token must have [permission](../user/permissions.md) to create pipelines in the project. @@ -473,6 +490,9 @@ and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with ap - If you run external PostgreSQL, particularly AWS RDS, [check you have a PostgreSQL bug fix](#postgresql-segmentation-fault-issue) to avoid the database crashing. +- The use of encrypted S3 buckets with storage-specific configuration is no longer supported after [removing support for using `background_upload`](removals.md#background-upload-for-object-storage). +- The [certificate-based Kubernetes integration (DEPRECATED)](../user/infrastructure/clusters/index.md#certificate-based-kubernetes-integration-deprecated) is disabled by default, but you can be re-enable it through the [`certificate_based_clusters` feature flag](../administration/feature_flags.md#how-to-enable-and-disable-features-behind-flags) until GitLab 16.0. +- When you use the GitLab Helm Chart project with a custom `serviceAccount`, ensure it has `get` and `list` permissions for the `serviceAccount` and `secret` resources. ### 14.10.0 @@ -634,7 +654,7 @@ or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [follo All merge request diff commits automatically incorporate these changes, and there are no additional requirements to perform the upgrade. Existing data in the `merge_request_diff_commits` table remains unpacked until you run `VACUUM FULL merge_request_diff_commits`. - But note that the `VACUUM FULL` operation locks and rewrites the entire `merge_request_diff_commits` table, + However, the `VACUUM FULL` operation locks and rewrites the entire `merge_request_diff_commits` table, so the operation takes some time to complete and it blocks access to this table until the end of the process. We advise you to only run this command while GitLab is not actively used or it is taken offline for the duration of the process. The time it takes to complete depends on the size of the table, which can be obtained by using `select pg_size_pretty(pg_total_relation_size('merge_request_diff_commits'));`. @@ -684,6 +704,10 @@ or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [follo as Sidekiq would continue using a bad connection. Geo and other features that rely on cron jobs running regularly do not work until Sidekiq is restarted. We recommend upgrading to GitLab 14.4.3 and later if this issue affects you. +- After enabling database load balancing by default in 14.4.0, we found an issue where + [Database load balancing does not work with an AWS Aurora cluster](https://gitlab.com/gitlab-org/gitlab/-/issues/220617). + We recommend moving your databases from Aurora to RDS for PostgreSQL before + upgrading. Refer to [Moving GitLab databases to a different PostgreSQL instance](../administration/postgresql/moving.md). - GitLab 14.4.0 includes a [background migration `PopulateTopicsTotalProjectsCountCache`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71033) that may remain stuck permanently in a **pending** state when the instance lacks records that match the migration's target. @@ -750,7 +774,7 @@ for how to proceed. - [`geo_job_artifact_deleted_events`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66763) - [`push_event_payloads`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67299) - `ci_job_artifacts`: - - [Finalize job_id conversion to `bigint` for `ci_job_artifacts`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67774) + - [Finalize `job_id` conversion to `bigint` for `ci_job_artifacts`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67774) - [Finalize `ci_job_artifacts` conversion to `bigint`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65601) If the migrations are executed as part of a no-downtime deployment, there's a risk of failure due to lock conflicts with the application logic, resulting in lock timeout or deadlocks. In each case, these migrations are safe to re-run until successful: @@ -992,10 +1016,10 @@ GitLab 13.2.0 [remediates](https://gitlab.com/gitlab-org/gitlab/-/merge_requests After upgrading, if some of your users are unexpectedly encountering 404 or 422 errors when signing in, or "blocked" messages when using the command line, their accounts may have been un-confirmed. -In that case, please ask them to check their email for a re-confirmation link. +In that case, ask them to check their email for a re-confirmation link. For more information, see our discussion of [Email confirmation issues](../user/upgrade_email_bypass.md). -GitLab 13.2.0 relies on the `btree_gist` extension for PostgreSQL. For installations with an externally managed PostgreSQL setup, please make sure to +GitLab 13.2.0 relies on the `btree_gist` extension for PostgreSQL. For installations with an externally managed PostgreSQL setup, make sure to [install the extension manually](https://www.postgresql.org/docs/11/sql-createextension.html) before upgrading GitLab if the database user for GitLab is not a superuser. This is not necessary for installations using a GitLab managed PostgreSQL database. @@ -1042,7 +1066,7 @@ If you persist your own Rack Attack initializers between upgrades, you might - The final patch release (12.10.14) [has a regression affecting maven package uploads](https://about.gitlab.com/releases/2020/07/06/critical-security-release-gitlab-13-1-3-released/#maven-package-upload-broken-in-121014). - If you use this feature and need to stay on 12.10 while preparing to upgrade to 13.0: + If you use this feature and must stay on 12.10 while preparing to upgrade to 13.0: - Upgrade to 12.10.13 instead. - Upgrade to 13.0.14 as soon as possible. @@ -1103,7 +1127,7 @@ When Geo is enabled, LFS objects fail to be saved for imported or mirrored proje ### PostgreSQL segmentation fault issue If you run GitLab with external PostgreSQL, particularly AWS RDS, ensure you upgrade PostgreSQL -to patch levels to a minimum of 12.10 or 13.3 before upgrading to GitLab 14.8 or later. +to patch levels to a minimum of 12.7 or 13.3 before upgrading to GitLab 14.8 or later. [In 14.8](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/75511) for GitLab Enterprise Edition and [in 15.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87983) diff --git a/doc/update/package/convert_to_ee.md b/doc/update/package/convert_to_ee.md index 34bd8e61107..9f8e56c460c 100644 --- a/doc/update/package/convert_to_ee.md +++ b/doc/update/package/convert_to_ee.md @@ -15,7 +15,7 @@ Converting from the same version of CE to EE is not explicitly necessary, and an you are upgrading the same version (for example, CE 12.1 to EE 12.1), which is **recommended**. WARNING: -When updating to EE from CE, avoid reverting back to CE if you plan on going to EE again in the +When updating to EE from CE, avoid reverting back to CE if you plan to go to EE again in the future. Reverting back to CE can cause [database issues](index.md#500-error-when-accessing-project--settings--repository) that may require Support intervention. @@ -31,7 +31,7 @@ The steps can be summed up to: ``` The output should be similar to: `Installed: 13.0.4-ce.0`. In that case, - the equivalent Enterprise Edition version will be: `13.0.4-ee.0`. Write this + the equivalent Enterprise Edition version is: `13.0.4-ee.0`. Write this value down. **For CentOS/RHEL** @@ -41,7 +41,7 @@ The steps can be summed up to: ``` The output should be similar to: `gitlab-ce-13.0.4-ce.0.el8.x86_64`. In that - case, the equivalent Enterprise Edition version will be: + case, the equivalent Enterprise Edition version is: `gitlab-ee-13.0.4-ee.0.el8.x86_64`. Write this value down. 1. Add the `gitlab-ee` [Apt or Yum repository](https://packages.gitlab.com/gitlab/gitlab-ee/install): @@ -58,13 +58,19 @@ The steps can be summed up to: curl --silent "https://packages.gitlab.com/install/repositories/gitlab/gitlab-ee/script.rpm.sh" | sudo bash ``` - The above command will find your OS version and automatically set up the + The above command finds your OS version and automatically set up the repository. If you are not comfortable installing the repository through a piped script, you can first [check its contents](https://packages.gitlab.com/gitlab/gitlab-ee/install). -1. Next, install the `gitlab-ee` package. Note that this will automatically - uninstall the `gitlab-ce` package on your GitLab server. `reconfigure` + NOTE: + If you want to use `dpkg`/`rpm` instead of `apt-get`/`yum`, go through the first + step to find the current GitLab version, then follow + [Update using a manually-downloaded package](index.md#upgrade-using-a-manually-downloaded-package), + and then [add your license](../../user/admin_area/license.md). + +1. Install the `gitlab-ee` package. The install automatically + uninstalls the `gitlab-ce` package on your GitLab server. `reconfigure` Omnibus right after the `gitlab-ee` package is installed. **Make sure that you install the exact same GitLab version**: @@ -91,8 +97,7 @@ The steps can be summed up to: sudo gitlab-ctl reconfigure ``` -1. Now go to the GitLab Admin Area of your server (`/admin/subscription`) and - [add your license](../../user/admin_area/license.md). +1. Now activate GitLab Enterprise Edition by [adding your license](../../user/admin_area/license.md). 1. After you confirm that GitLab is working as expected, you may remove the old Community Edition repository: @@ -111,8 +116,3 @@ The steps can be summed up to: That's it! You can now use GitLab Enterprise Edition! To update to a newer version, follow [Update using the official repositories](index.md#upgrade-using-the-official-repositories). - -NOTE: -If you want to use `dpkg`/`rpm` instead of `apt-get`/`yum`, go through the first -step to find the current GitLab version and then follow -[Update using a manually-downloaded package](index.md#upgrade-using-a-manually-downloaded-package). diff --git a/doc/update/package/downgrade.md b/doc/update/package/downgrade.md index 7b8b4da0383..f48ba31568f 100644 --- a/doc/update/package/downgrade.md +++ b/doc/update/package/downgrade.md @@ -79,5 +79,5 @@ Steps: sudo gitlab-ctl reconfigure ``` -1. [Restore GitLab](../../raketasks/backup_restore.md#restore-for-omnibus-gitlab-installations) +1. [Restore GitLab](../../raketasks/restore_gitlab.md#restore-for-omnibus-gitlab-installations) to complete the downgrade. diff --git a/doc/update/package/index.md b/doc/update/package/index.md index 15f43f59425..bf1154d1cf5 100644 --- a/doc/update/package/index.md +++ b/doc/update/package/index.md @@ -189,6 +189,11 @@ NOTE: For the GitLab Community Edition, replace `gitlab-ee` with `gitlab-ce`. +## Upgrade the product documentation + +This is an optional step. If you [installed the product documentation](../../administration/docs_self_host.md), +see how to [upgrade to a later version](../../administration/docs_self_host.md#upgrade-using-docker). + ## Troubleshooting ### GitLab 13.7 and later unavailable on Amazon Linux 2 @@ -300,7 +305,7 @@ To fix this issue: ### Error `Failed to connect to the internal GitLab API` on a separate GitLab Pages server -Please see [GitLab Pages troubleshooting](../../administration/pages/index.md#failed-to-connect-to-the-internal-gitlab-api). +See [GitLab Pages troubleshooting](../../administration/pages/index.md#failed-to-connect-to-the-internal-gitlab-api). ### Error `An error occurred during the signature verification` when running `apt-get update` diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index dc09f6063c8..b7c148045bd 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -11,11 +11,11 @@ comments: false Make sure you view [this update guide](https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/update/patch_versions.md) from the tag (version) of GitLab you would like to install. In most cases this should be the highest numbered production tag (without `rc` in it). -You can select the tag in the version dropdown in the top left corner of GitLab (below the menu bar). +You can select the tag in the version dropdown list in the top left corner of GitLab (below the menu bar). ### 0. Backup -It's useful to make a backup just in case things go south. Depending on the installation method, backup commands vary. See the [backing up and restoring GitLab](../raketasks/backup_restore.md) documentation. +Make a backup just in case things go south. Depending on the installation method, backup commands vary. See the [backing up and restoring GitLab](../raketasks/backup_restore.md) documentation. ### 1. Stop server @@ -107,7 +107,7 @@ sudo -u git -H make ### 8. Install/Update `gitlab-elasticsearch-indexer` **(PREMIUM SELF)** -Please follow the [install instruction](../integration/advanced_search/elasticsearch.md#install-elasticsearch). +Follow the [install instruction](../integration/advanced_search/elasticsearch.md#install-elasticsearch). ### 9. Start application diff --git a/doc/update/plan_your_upgrade.md b/doc/update/plan_your_upgrade.md index cdc3ec39f9a..2374856ff0c 100644 --- a/doc/update/plan_your_upgrade.md +++ b/doc/update/plan_your_upgrade.md @@ -87,7 +87,7 @@ to roll back GitLab to a working state if there's a problem with the upgrade: - Create a [GitLab backup](../raketasks/backup_restore.md). Make sure to follow the instructions based on your installation method. - Don't forget to back up the [secrets and configuration files](../raketasks/backup_restore.md#storing-configuration-files). + Don't forget to back up the [secrets and configuration files](../raketasks/backup_gitlab.md#storing-configuration-files). - Alternatively, create a snapshot of your instance. If this is a multi-node installation, you must snapshot every node. **This process is out of scope for GitLab Support.** @@ -103,7 +103,7 @@ To restore your GitLab backup: the versions of the backed up and the new GitLab instance must be the same. - [Restore GitLab](../raketasks/backup_restore.md#restore-gitlab). Make sure to follow the instructions based on your installation method. - Confirm that the [secrets and configuration files](../raketasks/backup_restore.md#storing-configuration-files) are also restored. + Confirm that the [secrets and configuration files](../raketasks/backup_gitlab.md#storing-configuration-files) are also restored. - If restoring from a snapshot, know the steps to do this. **This process is out of scope for GitLab Support.** diff --git a/doc/update/removals.md b/doc/update/removals.md index 299d1b4c341..fa5c016d3ab 100644 --- a/doc/update/removals.md +++ b/doc/update/removals.md @@ -31,6 +31,25 @@ For removal reviewers (Technical Writers only): https://about.gitlab.com/handbook/marketing/blog/release-posts/#update-the-removals-doc --> +## Removed in 15.2 + +### Support for older browsers + +In GitLab 15.2, we are cleaning up and [removing old code](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86003) that was specific for browsers that we no longer support. This has no impact on users if they use one of our [supported web browsers](https://docs.gitlab.com/ee/install/requirements.html#supported-web-browsers). + +Most notably, support for the following browsers has been removed: + +- Apple Safari 14 and older. +- Mozilla Firefox 78. + +The minimum supported browser versions are: + +- Apple Safari 14.1. +- Mozilla Firefox 91. +- Google Chrome 92. +- Chromium 92. +- Microsoft Edge 92. + ## Removed in 15.0 ### API: `stale` status returned instead of `offline` or `not_connected` @@ -66,14 +85,45 @@ This is a [breaking change](https://docs.gitlab.com/ee/development/contributing/ Review the details carefully before upgrading. To reduce the overall complexity and maintenance burden of GitLab's [object storage feature](https://docs.gitlab.com/ee/administration/object_storage.html), support for using `background_upload` has been removed in GitLab 15.0. +By default [direct upload](https://docs.gitlab.com/ee/development/uploads/index.html#direct-upload) will be used. -This impacts a small subset of object storage providers, including but not limited to: +This impacts a subset of object storage providers, including but not limited to: - **OpenStack** Customers using OpenStack need to change their configuration to use the S3 API instead of Swift. - **RackSpace** Customers using RackSpace-based object storage need to migrate data to a different provider. If your object storage provider does not support `background_upload`, please [migrate objects to a supported object storage provider](https://docs.gitlab.com/ee/administration/object_storage.html#migrate-objects-to-a-different-object-storage-provider). +#### Encrypted S3 buckets + +Additionally, this also breaks the use of [encrypted S3 buckets](https://docs.gitlab.com/ee/administration/object_storage.html#encrypted-s3-buckets) with [storage-specific configuration form](https://docs.gitlab.com/ee/administration/object_storage.html#storage-specific-configuration). + +If your S3 buckets have [SSE-S3 or SSE-KMS encryption enabled](https://docs.aws.amazon.com/kms/latest/developerguide/services-s3.html), please [migrate your configuration to use consolidated object storage form](https://docs.gitlab.com/ee/administration/object_storage.html#transition-to-consolidated-form) before upgrading to GitLab 15.0. Otherwise, you may start getting `ETag mismatch` errors during objects upload. + +#### 403 errors + +If you see 403 errors when uploading to object storage after +upgrading to GitLab 15.0, check that the [correct permissions](https://docs.gitlab.com/ee/administration/object_storage.html#iam-permissions) +are assigned to the bucket. Direct upload needs the ability to delete an +object (example: `s3:DeleteObject`), but background uploads do not. + +#### `remote_directory` with a path prefix + +If the object storage `remote_directory` configuration contains a slash (`/`) after the bucket (example: `gitlab/uploads`), be aware that this [was never officially supported](https://gitlab.com/gitlab-org/gitlab/-/issues/292958). +Some users found that they could specify a path prefix to the bucket. In direct upload mode, object storage uploads will fail if a slash is present in GitLab 15.0. + +If you have set a prefix, you can use a workaround to revert to background uploads: + +1. Continue to use [storage-specific configuration](https://docs.gitlab.com/ee/administration/object_storage.html#storage-specific-configuration). +1. In Omnibus GitLab, set the `GITLAB_LEGACY_BACKGROUND_UPLOADS` to re-enable background uploads: + + ```ruby + gitlab_rails['env'] = { 'GITLAB_LEGACY_BACKGROUND_UPLOADS' => 'artifacts,external_diffs,lfs,uploads,packages,dependency_proxy,terraform_state,pages' } + ``` + +Prefixes will be supported officially in [GitLab 15.2](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91307). +This workaround will be dropped, so we encourage migrating to consolidated object storage. + ### Container Network and Host Security WARNING: @@ -443,6 +493,22 @@ You can still customize the behavior of the Secret Detection analyzer using the For further details, see [the deprecation issue for this change](https://gitlab.com/gitlab-org/gitlab/-/issues/352565). +### Self-managed certificate-based integration with Kubernetes feature flagged + +WARNING: +This is a [breaking change](https://docs.gitlab.com/ee/development/contributing/#breaking-changes). +Review the details carefully before upgrading. + +In 15.0 the certificate-based integration with Kubernetes will be disabled by default. + +After 15.0, you should use the [agent for Kubernetes](https://docs.gitlab.com/ee/user/clusters/agent/) to connect Kubernetes clusters with GitLab. The agent for Kubernetes is a more robust, secure, and reliable integration with Kubernetes. [How do I migrate to the agent?](https://docs.gitlab.com/ee/user/infrastructure/clusters/migrate_to_gitlab_agent.html) + +If you need more time to migrate, you can enable the `certificate_based_clusters` [feature flag](https://docs.gitlab.com/ee/administration/feature_flags.html), which re-enables the certificate-based integration. + +In GitLab 16.0, we will [remove the feature, its related code, and the feature flag](https://about.gitlab.com/blog/2021/11/15/deprecating-the-cert-based-kubernetes-integration/). GitLab will continue to fix any security or critical issues until 16.0. + +For updates and details, follow [this epic](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). + ### Sidekiq configuration for metrics and health checks WARNING: @@ -553,9 +619,9 @@ Review the details carefully before upgrading. The `Managed-Cluster-Applications.gitlab-ci.yml` CI/CD template is being removed. If you need an alternative, try the [Cluster Management project template](https://gitlab.com/gitlab-org/gitlab/-/issues/333610) instead. If your are not ready to move, you can copy the [last released version](https://gitlab.com/gitlab-org/gitlab-foss/-/blob/v14.10.1/lib/gitlab/ci/templates/Managed-Cluster-Applications.gitlab-ci.yml) of the template into your project. -### `artifacts:report:cobertura` keyword +### `artifacts:reports:cobertura` keyword -As of GitLab 15.0, the [`artifacts:report:cobertura`](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscobertura-removed) +As of GitLab 15.0, the [`artifacts:reports:cobertura`](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscobertura-removed) keyword has been [replaced](https://gitlab.com/gitlab-org/gitlab/-/issues/344533) by [`artifacts:reports:coverage_report`](https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscoverage_report). Cobertura is the only supported report file, but this is the first step towards GitLab supporting other report types. diff --git a/doc/update/upgrading_from_ce_to_ee.md b/doc/update/upgrading_from_ce_to_ee.md index d6548620356..b953194b4cf 100644 --- a/doc/update/upgrading_from_ce_to_ee.md +++ b/doc/update/upgrading_from_ce_to_ee.md @@ -21,7 +21,7 @@ GitLab edition you are using (Community or Enterprise), see the This guide assumes you have a correctly configured and tested installation of GitLab Community Edition. If you run into any trouble or if you have any -questions please contact us at `support@gitlab.com`. +questions contact us at `support@gitlab.com`. In all examples, replace `EE_BRANCH` with the Enterprise Edition branch for the version you are using, and `CE_BRANCH` with the Community Edition branch. @@ -41,7 +41,7 @@ cd /home/git/gitlab sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ``` -For installations using MySQL, this may require granting "LOCK TABLES" +For installations using MySQL, this may require granting `LOCK TABLES` privileges to the GitLab user on the database version. ### 1. Stop server @@ -88,7 +88,7 @@ sudo -u git -H bundle exec rake cache:clear RAILS_ENV=production ### 4. Install `gitlab-elasticsearch-indexer` **(PREMIUM SELF)** -Please follow the [install instruction](../integration/advanced_search/elasticsearch.md#install-elasticsearch). +Follow the [install instruction](../integration/advanced_search/elasticsearch.md#install-elasticsearch). ### 5. Start application diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index 29bb956cb54..8b921f6d0ce 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -8,12 +8,12 @@ comments: false # Upgrading Community Edition and Enterprise Edition from source **(FREE SELF)** Make sure you view this update guide from the branch (version) of GitLab you -would like to install (for example, `11.8`). You can select the required version of documentation in the dropdown at the top right corner of GitLab documentation page. +would like to install (for example, `11.8`). You can select the required version of documentation in the dropdown list at the top right corner of GitLab documentation page. In each of the following examples, replace `BRANCH` with the branch of the version you upgrading to (for example, `11-8-stable` for `11.8`). Replace `PREVIOUS_BRANCH` with the branch for the version you are upgrading from (for example, `11-7-stable` for `11.7`). -If the highest number stable branch is unclear please check the +If the highest number stable branch is unclear check the [GitLab Blog](https://about.gitlab.com/blog/archives.html) for installation guide links by version. @@ -24,7 +24,7 @@ the [Upgrading from CE to EE](upgrading_from_ce_to_ee.md) documentation. Major versions are reserved for backwards incompatible changes. We recommend that you first upgrade to the latest available minor version of your current major version. -Please follow the [Upgrade Recommendations](../policy/maintenance.md#upgrade-recommendations) +Follow the [Upgrade Recommendations](../policy/maintenance.md#upgrade-recommendations) to identify the ideal upgrade path. Before upgrading to a new major version, you should ensure that any background @@ -225,7 +225,7 @@ NGINX configuration to continue using it. This is because the GitLab application sets it. If you are using Apache instead of NGINX see the updated [Apache templates](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/web-server/apache). -Also note that because Apache does not support upstreams behind Unix sockets you +Also because Apache does not support upstreams behind Unix sockets you must let GitLab Workhorse listen on a TCP port. You can do this via [`/etc/default/gitlab`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/init.d/gitlab.default.example#L38). @@ -406,6 +406,11 @@ sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production If all items are green, then congratulations, the upgrade is complete! +### 17. Upgrade the product documentation + +This is an optional step. If you [installed the product documentation](../install/installation.md#install-the-product-documentation), +see how to [upgrade to a later version](../administration/docs_self_host.md#upgrade-the-product-documentation-to-a-later-version). + ## Version specific upgrading instructions This section contains upgrading instructions for specific versions. When @@ -423,7 +428,7 @@ Additional instructions here. ### 15.0.0 Support for more than one database has been added to GitLab. [As part of this](https://gitlab.com/gitlab-org/gitlab/-/issues/338182), -`config/database.yml` needs to include a database name in the database configuration. +`config/database.yml` must include a database name in the database configuration. The `main: database` must be first. If an invalid or deprecated syntax is used, an error is generated during application start: @@ -442,7 +447,7 @@ production: ... ``` -Starting with GitLab 15.0, it needs to define a `main` database first: +Starting with GitLab 15.0, it must define a `main` database first: ```yaml production: @@ -487,7 +492,7 @@ for the previous version. For example, if you have upgraded to GitLab 12.6 and want to revert back to 12.5, follow the guides for upgrading from 12.4 to 12.5. You can -use the version dropdown at the top of the page to select the right version. +use the version dropdown list at the top of the page to select the right version. When reverting, you should **not** follow the database migration guides, as the backup has already been migrated to the previous version. diff --git a/doc/update/with_downtime.md b/doc/update/with_downtime.md index 9357f70e44a..c6ad854fd0f 100644 --- a/doc/update/with_downtime.md +++ b/doc/update/with_downtime.md @@ -15,12 +15,12 @@ there are a number of constraints. In particular, you can upgrade to only one mi at a time, for example, from 14.6 to 14.7, then to 14.8, etc. If you want to upgrade to more than one minor release at a time (for example, from 14.6 to 14.9), -you need to take your GitLab instance offline, which implies downtime. +you must take your GitLab instance offline, which implies downtime. Before starting this process, verify the [version specific upgrading instructions](index.md#version-specific-upgrading-instructions) relevant to your [upgrade path](index.md#upgrade-paths). -For a single node installation, you only need to [uprgade the GitLab package](package/index.md). +For a single node installation, you must only [upgrade the GitLab package](package/index.md). The process for upgrading a number of components of a multi-node GitLab installation is the same as for zero downtime upgrades. @@ -86,11 +86,11 @@ for Gitaly cluster. If you are using Amazon Machine Images (AMIs) on AWS, the Gitaly nodes **should not be upgraded via the AMI process**. Gitaly nodes should **only** -be upgraded using the package upgrade. This is because: +be upgraded using the package upgrade because: - Praefect tracks replicas of Git repositories by server hostname. -- Redeployment using AMIs will issue the nodes with new hostnames. -- Even though the storage will be the same, Gitaly cluster will not work after this. +- Redeployment using AMIs issues the nodes with new hostnames. +- Even though the storage is the same, Gitaly cluster does not work after this. The Praefect nodes, however, can be upgraded via an AMI redeployment process: @@ -100,7 +100,7 @@ The Praefect nodes, however, can be upgraded via an AMI redeployment process: 1. The first node to be redeployed with the upgraded image should be your deploy node. 1. After it's deployed, set `praefect['auto_migrate'] = true` in `gitlab.rb` - and apply with `gitlab-ctl reconfigure`. This will run the database + and apply with `gitlab-ctl reconfigure`. This runs the database migrations. 1. Redeploy your other Praefect nodes. @@ -142,7 +142,7 @@ For unclustered PostgreSQL servers: ## Upgrade the Patroni node -Patroni is used to achiece high availabilty with PostgreSQL. +Patroni is used to achieve high availability with PostgreSQL. If a PostgreSQL major version upgrade is required, [follow the major version process](../administration/postgresql/replication_and_failover.md#upgrading-postgresql-major-version-in-a-patroni-cluster). @@ -150,7 +150,7 @@ If a PostgreSQL major version upgrade is required, The upgrade process for all other versions is performed on all replicas first. After they're upgraded, a cluster failover occurs from the leader to one of the upgraded replicas. This ensures that only one failover is needed, and once complete the new -leader will be upgraded. +leader is upgraded. Follow the following process: @@ -218,7 +218,7 @@ sudo yum install gitlab-ee ## Upgrade Redis HA (using Sentinel) **(PREMIUM SELF)** -Follow [the zero downtime instructions](zero_downtime.md#use-redis-ha-using-sentinel) +Follow [the zero downtime instructions](zero_downtime.md#redis-ha-using-sentinel) for upgrading your Redis HA cluster. ## Upgrade the Rails nodes (Puma / Sidekiq) @@ -232,7 +232,7 @@ All the Puma and Sidekiq processes were previously shut down. On each node: ps -ef | egrep 'puma: | puma | sidekiq ' ``` -Select one node that runs Puma. This will be your deploy node, and is responsible for +Select one node that runs Puma. This is your deploy node, and is responsible for running all database migrations. On the deploy node: 1. Ensure the server is configured to permit regular migrations. Check that diff --git a/doc/update/zero_downtime.md b/doc/update/zero_downtime.md index a3f9886ed0b..3cdc6177a4d 100644 --- a/doc/update/zero_downtime.md +++ b/doc/update/zero_downtime.md @@ -27,14 +27,14 @@ If you meet all the requirements above, follow these instructions in order. Ther | Deployment type | Description | | --------------------------------------------------------------- | ------------------------------------------------ | | [Gitaly or Gitaly Cluster](#gitaly-or-gitaly-cluster) | GitLab CE/EE using HA architecture for Gitaly or Gitaly Cluster | -| [Multi-node / PostgreSQL HA](#use-postgresql-ha) | GitLab CE/EE using HA architecture for PostgreSQL | -| [Multi-node / Redis HA](#use-redis-ha-using-sentinel) | GitLab CE/EE using HA architecture for Redis | +| [Multi-node / PostgreSQL HA](#postgresql) | GitLab CE/EE using HA architecture for PostgreSQL | +| [Multi-node / Redis HA](#redis-ha-using-sentinel) | GitLab CE/EE using HA architecture for Redis | | [Geo](#geo-deployment) | GitLab EE with Geo enabled | | [Multi-node / HA with Geo](#multi-node--ha-deployment-with-geo) | GitLab CE/EE on multiple nodes | Each type of deployment requires that you hot reload the `puma` and `sidekiq` processes on all nodes running these services after you've upgraded. The reason for this is that those processes each load the GitLab Rails application which reads and loads -the database schema into memory when starting up. Each of these processes needs to be reloaded (or restarted in the case of `sidekiq`) +the database schema into memory when starting up. Each of these processes must be reloaded (or restarted in the case of `sidekiq`) to re-read any database changes that have been made by post-deployment migrations. Most of the time you can safely upgrade from a patch release to the next minor @@ -260,7 +260,7 @@ node first and run database migrations. sudo gitlab-ctl reconfigure ``` -### Use PostgreSQL HA +### PostgreSQL Pick a node to be the `Deploy Node`. It can be any application node, but it must be the same node throughout the process. @@ -277,7 +277,7 @@ node throughout the process. - To prevent `reconfigure` from automatically running database migrations, ensure that `gitlab_rails['auto_migrate'] = false` is set in `/etc/gitlab/gitlab.rb`. -**Gitaly only nodes** +**PostgreSQL only nodes** - Update the GitLab package @@ -313,7 +313,7 @@ node throughout the process. - If you're using PgBouncer: - You need to bypass PgBouncer and connect directly to the database leader + You must bypass PgBouncer and connect directly to the database leader before running migrations. Rails uses an advisory lock when attempting to run a migration to prevent @@ -385,7 +385,7 @@ sure you remove `/etc/gitlab/skip-auto-reconfigure` and revert setting `gitlab_rails['auto_migrate'] = false` in `/etc/gitlab/gitlab.rb` after you've completed these steps. -### Use Redis HA (using Sentinel) **(PREMIUM SELF)** +### Redis HA (using Sentinel) **(PREMIUM SELF)** Package upgrades may involve version updates to the bundled Redis service. On instances using [Redis for scaling](../administration/redis/index.md), @@ -395,11 +395,11 @@ HA. #### In the application node -According to [official Redis docs](https://redis.io/topics/admin#upgrading-or-restarting-a-redis-instance-without-downtime), +According to [official Redis documentation](https://redis.io/topics/admin#upgrading-or-restarting-a-redis-instance-without-downtime), the easiest way to update an HA instance using Sentinel is to upgrade the secondaries one after the other, perform a manual failover from current primary (running old version) to a recently upgraded secondary (running a new -version), and then upgrade the original primary. For this, we need to know +version), and then upgrade the original primary. For this, we must know the address of the current Redis primary. - If your application node is running GitLab 12.7.0 or later, you can use the @@ -416,7 +416,7 @@ following command to get address of current Redis primary 1. Get the address of one of the sentinel nodes specified as `gitlab_rails['redis_sentinels']` in `/etc/gitlab/gitlab.rb` - 1. Get the Redis master name specified as `redis['master_name']` in + 1. Get the Redis main name specified as `redis['master_name']` in `/etc/gitlab/gitlab.rb` 1. Run the following command @@ -427,6 +427,8 @@ following command to get address of current Redis primary #### In the Redis secondary nodes +1. Set `gitlab_rails['rake_cache_clear'] = false` in `gitlab.rb` if you haven't already. If not, you might receive the error `Redis::CommandError: READONLY You can't write against a read only replica.` during the reconfigure post installation of new package. + 1. Install package for new version. 1. Run `sudo gitlab-ctl reconfigure`, if a reconfigure is not run as part of @@ -442,8 +444,8 @@ following command to get address of current Redis primary #### In the Redis primary node -Before upgrading the Redis primary node, we need to perform a failover so that -one of the recently upgraded secondary nodes becomes the new primary. Once the +Before upgrading the Redis primary node, we must perform a failover so that +one of the recently upgraded secondary nodes becomes the new primary. After the failover is complete, we can go ahead and upgrade the original primary node. 1. Stop Redis service in Redis primary node so that it fails over to a secondary @@ -623,7 +625,7 @@ Updates must be performed in the following order: ### Step 1: Choose a "deploy node" for each deployment -You now need to choose: +You now must choose: - One instance for use as the **primary** "deploy node" on the Geo **primary** multi-node deployment. - One instance for use as the **secondary** "deploy node" on each Geo **secondary** multi-node deployment. @@ -696,7 +698,7 @@ sudo touch /etc/gitlab/skip-auto-reconfigure 1. If you're using PgBouncer: - You need to bypass PgBouncer and connect directly to the database leader + You must bypass PgBouncer and connect directly to the database leader before running migrations. Rails uses an advisory lock when attempting to run a migration to prevent diff --git a/doc/user/admin_area/analytics/dev_ops_report.md b/doc/user/admin_area/analytics/dev_ops_report.md deleted file mode 100644 index 077718863e7..00000000000 --- a/doc/user/admin_area/analytics/dev_ops_report.md +++ /dev/null @@ -1,9 +0,0 @@ ---- -redirect_to: 'dev_ops_reports.md' -remove_date: '2022-06-16' ---- - -This document was moved to [another location](dev_ops_reports.md). - -<!-- This redirect file can be deleted after <2022-06-16>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/admin_area/broadcast_messages.md b/doc/user/admin_area/broadcast_messages.md index 9d4c1ffe375..959331c16de 100644 --- a/doc/user/admin_area/broadcast_messages.md +++ b/doc/user/admin_area/broadcast_messages.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Activation +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference, howto --- diff --git a/doc/user/admin_area/geo_nodes.md b/doc/user/admin_area/geo_nodes.md index 3c33578b88f..710f37bb344 100644 --- a/doc/user/admin_area/geo_nodes.md +++ b/doc/user/admin_area/geo_nodes.md @@ -1,117 +1,11 @@ --- -stage: Systems -group: Geo -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: 'geo_sites.md' +remove_date: '2022-10-05' --- -# Geo sites Admin Area **(PREMIUM SELF)** +This document was moved to [another location](geo_sites.md). -You can configure various settings for GitLab Geo sites. For more information, see -[Geo documentation](../../administration/geo/index.md). - -On either the primary or secondary site: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Sites**. - -## Common settings - -All Geo sites have the following settings: - -| Setting | Description | -| --------| ----------- | -| Primary | This marks a Geo site as **primary** site. There can be only one **primary** site. | -| Name | The unique identifier for the Geo site. It's highly recommended to use a physical location as a name. Good examples are "London Office" or "us-east-1". Avoid words like "primary", "secondary", "Geo", or "DR". This makes the failover process easier because the physical location does not change, but the Geo site role can. All nodes in a single Geo site use the same site name. Nodes use the `gitlab_rails['geo_node_name']` setting in `/etc/gitlab/gitlab.rb` to lookup their Geo site record in the PostgreSQL database. If `gitlab_rails['geo_node_name']` is not set, the node's `external_url` with trailing slash is used as fallback. The value of `Name` is case-sensitive, and most characters are allowed. | -| URL | The instance's user-facing URL. | - -The site you're currently browsing is indicated with a blue `Current` label, and -the **primary** node is listed first as `Primary site`. - -## Secondary site settings - -**Secondary** sites have a number of additional settings available: - -| Setting | Description | -|---------------------------|-------------| -| Selective synchronization | Enable Geo [selective sync](../../administration/geo/replication/configuration.md#selective-synchronization) for this **secondary** site. | -| Repository sync capacity | Number of concurrent requests this **secondary** site makes to the **primary** site when backfilling repositories. | -| File sync capacity | Number of concurrent requests this **secondary** site makes to the **primary** site when backfilling files. | - -## Geo backfill - -**Secondary** sites are notified of changes to repositories and files by the **primary** site, -and always attempt to synchronize those changes as quickly as possible. - -Backfill is the act of populating the **secondary** site with repositories and files that -existed *before* the **secondary** site was added to the database. Because there may be -extremely large numbers of repositories and files, it's not feasible to attempt to -download them all at once; so, GitLab places an upper limit on the concurrency of -these operations. - -How long the backfill takes is dependent on the maximum concurrency, but higher -values place more strain on the **primary** site. The limits are configurable. -If your **primary** site has lots of surplus capacity, -you can increase the values to complete backfill in a shorter time. If it's -under heavy load and backfill reduces its availability for normal requests, -you can decrease them. - -## Set up the internal URLs - -> Setting up internal URLs in secondary sites was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77179) in GitLab 14.7. - -You can set up a different URL for synchronization between the primary and secondary site. - -The **primary** site's Internal URL is used by **secondary** sites to contact it -(to sync repositories, for example). The name Internal URL distinguishes it from -[External URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab), -which is used by users. Internal URL does not need to be a private address. - -When [Geo secondary proxying](../../administration/geo/secondary_proxy/index.md) is enabled, -the primary uses the secondary's internal URL to contact it directly. - -The internal URL defaults to external URL. To change it: - -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Geo > Nodes**. -1. Select **Edit** on the site you want to customize. -1. Edit the internal URL. -1. Select **Save changes**. - -When enabled, the Admin Area for Geo shows replication details for each site directly -from the primary site's UI, and through the Geo secondary proxy, if enabled. - -WARNING: -We recommend using an HTTPS connection while configuring the Geo sites. To avoid -breaking communication between **primary** and **secondary** sites when using -HTTPS, customize your Internal URL to point to a load balancer with TLS -terminated at the load balancer. - -WARNING: -Starting with GitLab 13.3 and [until 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/325522), -if you use an internal URL that is not accessible to the users, the -OAuth authorization flow does not work properly, because users are redirected -to the internal URL instead of the external one. - -## Multiple secondary sites behind a load balancer - -**Secondary** sites can use identical external URLs if -a unique `name` is set for each Geo site. The `gitlab.rb` setting -`gitlab_rails['geo_node_name']` must: - -- Be set for each GitLab instance that runs `puma`, `sidekiq`, or `geo_logcursor`. -- Match a Geo site name. - -The load balancer must use sticky sessions to avoid authentication -failures and cross-site request errors. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after <2022-10-05>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/admin_area/geo_sites.md b/doc/user/admin_area/geo_sites.md new file mode 100644 index 00000000000..e577fdf60f1 --- /dev/null +++ b/doc/user/admin_area/geo_sites.md @@ -0,0 +1,117 @@ +--- +stage: Systems +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Geo sites Admin Area **(PREMIUM SELF)** + +You can configure various settings for GitLab Geo sites. For more information, see +[Geo documentation](../../administration/geo/index.md). + +On either the primary or secondary site: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Geo > Sites**. + +## Common settings + +All Geo sites have the following settings: + +| Setting | Description | +| --------| ----------- | +| Primary | This marks a Geo site as **primary** site. There can be only one **primary** site. | +| Name | The unique identifier for the Geo site. It's highly recommended to use a physical location as a name. Good examples are "London Office" or "us-east-1". Avoid words like "primary", "secondary", "Geo", or "DR". This makes the failover process easier because the physical location does not change, but the Geo site role can. All nodes in a single Geo site use the same site name. Nodes use the `gitlab_rails['geo_node_name']` setting in `/etc/gitlab/gitlab.rb` to lookup their Geo site record in the PostgreSQL database. If `gitlab_rails['geo_node_name']` is not set, the node's `external_url` with trailing slash is used as fallback. The value of `Name` is case-sensitive, and most characters are allowed. | +| URL | The instance's user-facing URL. | + +The site you're currently browsing is indicated with a blue `Current` label, and +the **primary** node is listed first as `Primary site`. + +## Secondary site settings + +**Secondary** sites have a number of additional settings available: + +| Setting | Description | +|---------------------------|-------------| +| Selective synchronization | Enable Geo [selective sync](../../administration/geo/replication/configuration.md#selective-synchronization) for this **secondary** site. | +| Repository sync capacity | Number of concurrent requests this **secondary** site makes to the **primary** site when backfilling repositories. | +| File sync capacity | Number of concurrent requests this **secondary** site makes to the **primary** site when backfilling files. | + +## Geo backfill + +**Secondary** sites are notified of changes to repositories and files by the **primary** site, +and always attempt to synchronize those changes as quickly as possible. + +Backfill is the act of populating the **secondary** site with repositories and files that +existed *before* the **secondary** site was added to the database. Because there may be +extremely large numbers of repositories and files, it's not feasible to attempt to +download them all at once; so, GitLab places an upper limit on the concurrency of +these operations. + +How long the backfill takes is dependent on the maximum concurrency, but higher +values place more strain on the **primary** site. The limits are configurable. +If your **primary** site has lots of surplus capacity, +you can increase the values to complete backfill in a shorter time. If it's +under heavy load and backfill reduces its availability for normal requests, +you can decrease them. + +## Set up the internal URLs + +> Setting up internal URLs in secondary sites was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/77179) in GitLab 14.7. + +You can set up a different URL for synchronization between the primary and secondary site. + +The **primary** site's Internal URL is used by **secondary** sites to contact it +(to sync repositories, for example). The name Internal URL distinguishes it from +[External URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab), +which is used by users. Internal URL does not need to be a private address. + +When [Geo secondary proxying](../../administration/geo/secondary_proxy/index.md) is enabled, +the primary uses the secondary's internal URL to contact it directly. + +The internal URL defaults to external URL. To change it: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Geo > Sites**. +1. Select **Edit** on the site you want to customize. +1. Edit the internal URL. +1. Select **Save changes**. + +When enabled, the Admin Area for Geo shows replication details for each site directly +from the primary site's UI, and through the Geo secondary proxy, if enabled. + +WARNING: +We recommend using an HTTPS connection while configuring the Geo sites. To avoid +breaking communication between **primary** and **secondary** sites when using +HTTPS, customize your Internal URL to point to a load balancer with TLS +terminated at the load balancer. + +WARNING: +Starting with GitLab 13.3 and [until 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/325522), +if you use an internal URL that is not accessible to the users, the +OAuth authorization flow does not work properly, because users are redirected +to the internal URL instead of the external one. + +## Multiple secondary sites behind a load balancer + +**Secondary** sites can use identical external URLs if +a unique `name` is set for each Geo site. The `gitlab.rb` setting +`gitlab_rails['geo_node_name']` must: + +- Be set for each GitLab instance that runs `puma`, `sidekiq`, or `geo_logcursor`. +- Match a Geo site name. + +The load balancer must use sticky sessions to avoid authentication +failures and cross-site request errors. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 8f36021084e..5fd44cf8697 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -33,7 +33,7 @@ The Admin Area is made up of the following sections: | **{license}** License | Add, display, and remove [licenses](license.md). | | **{cloud-gear}** Kubernetes | Create and manage instance-level [Kubernetes clusters](../instance/clusters/index.md). | | **{push-rules}** Push rules | Configure pre-defined Git [push rules](../project/repository/push_rules.md) for projects. Also, configure [merge requests approvers rules](merge_requests_approvals.md). | -| **{location-dot}** Geo | Configure and maintain [Geo nodes](geo_nodes.md). | +| **{location-dot}** Geo | Configure and maintain [Geo sites](geo_sites.md). | | **{key}** Deploy keys | Create instance-wide [SSH deploy keys](../project/deploy_keys/index.md). | | **{lock}** Credentials | View [credentials](credentials_inventory.md) that can be used to access your instance. | | **{template}** Integrations | Manage [instance-level default settings](settings/project_integration_management.md) for a project integration. | @@ -184,7 +184,7 @@ The following data is included in the export: - Type - Path - Access level ([Project](../permissions.md#project-members-permissions) and [Group](../permissions.md#group-members-permissions)) -- Date of last activity ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345388) in GitLab 14.6). For a list of activities that populate this column, see the [Users API documentation](../../api/users.md#get-user-activities-administrator-only). +- Date of last activity ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345388) in GitLab 14.6). For a list of activities that populate this column, see the [Users API documentation](../../api/users.md#get-user-activities). Only the first 100,000 user accounts are exported. @@ -270,6 +270,8 @@ To create a new topic, select **New topic**. To edit a topic, select **Edit** in that topic's row. +To remove a topic, select **Remove** in that topic's row. + To search for topics by name, enter your criteria in the search box. The topic search is case insensitive and applies partial matching. @@ -370,14 +372,16 @@ The following topics document the **Monitoring** section of the Admin Area. ### System Information +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341248) in GitLab 15.2, support for relative time. "Uptime" statistic was renamed to "System started". + The **System Info** page provides the following statistics: -| Field | Description | -|:-------------|:------------| -| CPU | Number of CPU cores available | -| Memory Usage | Memory in use, and total memory available | -| Disk Usage | Disk space in use, and total disk space available | -| Uptime | Approximate uptime of the GitLab instance | +| Field | Description | +|:---------------|:--------------------------------------------------| +| CPU | Number of CPU cores available | +| Memory Usage | Memory in use, and total memory available | +| Disk Usage | Disk space in use, and total disk space available | +| System started | When the system hosting GitLab was started. In GitLab 15.1 and earlier, this was an uptime statistic. | These statistics are updated only when you navigate to the **System Info** page, or you refresh the page in your browser. diff --git a/doc/user/admin_area/license_file.md b/doc/user/admin_area/license_file.md index be1b1a16e29..99669b2a4d3 100644 --- a/doc/user/admin_area/license_file.md +++ b/doc/user/admin_area/license_file.md @@ -56,9 +56,10 @@ usage data monthly. To submit the data, [export your license usage](../../subscriptions/self_managed/index.md#export-your-license-usage) and send it by email to the renewals service, `renewals-service@customers.gitlab.com`. -If you don't submit your data each month after your subscription start date, a banner displays to remind you to -submit your data. The banner displays in the **Admin Area** on the **Dashboard** and on the **Subscription** -pages. You can only dismiss it until the following month after you submit your license usage data. +If you don't submit your data each month after your subscription start date, an email is sent to the address +associated with your subscription and a banner displays to remind you to submit your data. The banner displays +in the **Admin Area** on the **Dashboard** and on the **Subscription** pages. You can only dismiss it until the +following month after you submit your license usage data. ## What happens when your license expires diff --git a/doc/user/admin_area/moderate_users.md b/doc/user/admin_area/moderate_users.md index dc6ff96c31f..ab581cd3aa8 100644 --- a/doc/user/admin_area/moderate_users.md +++ b/doc/user/admin_area/moderate_users.md @@ -265,7 +265,7 @@ Use the Admin Area to delete users. 1. Select **Delete user**. NOTE: -You can only delete a user if there are inherited or direct owners of a group. You cannot delete a user if they are the only group owner. +You can only delete a user if there are inherited or direct owners of a group. You cannot delete a user if they are the only group owner. You can also delete a user and their contributions, such as merge requests, issues, and groups of which they are the only group owner. diff --git a/doc/user/admin_area/monitoring/background_migrations.md b/doc/user/admin_area/monitoring/background_migrations.md index 53d5056bb65..02d32099c63 100644 --- a/doc/user/admin_area/monitoring/background_migrations.md +++ b/doc/user/admin_area/monitoring/background_migrations.md @@ -28,6 +28,9 @@ You can [check the status of existing migrations](../../../update/index.md#batch ## Enable or disable batched background migrations +WARNING: +If you disable this feature flag, GitLab upgrades may fail. + Batched background migrations are under development but ready for production use. It is deployed behind a feature flag that is **enabled by default**. [GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) diff --git a/doc/user/admin_area/reporting/git_abuse_rate_limit.md b/doc/user/admin_area/reporting/git_abuse_rate_limit.md new file mode 100644 index 00000000000..ad3ecfa3a5a --- /dev/null +++ b/doc/user/admin_area/reporting/git_abuse_rate_limit.md @@ -0,0 +1,29 @@ +--- +stage: Anti-Abuse +group: Anti-Abuse +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Git abuse rate limit **(ULTIMATE SELF)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/8066) in GitLab 15.2 [with flags](../../../administration/feature_flags.md) named `git_abuse_rate_limit_feature_flag` and `auto_ban_user_on_excessive_projects_download`. Both flags are disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flags](../../../administration/feature_flags.md) named `git_abuse_rate_limit_feature_flag` and `auto_ban_user_on_excessive_projects_download`. + +Git abuse rate limiting is a feature to automatically [ban users](../moderate_users.md#ban-and-unban-users) who download more than a specified number of repositories in a given time. When the `git_abuse_rate_limit_feature_flag` feature flag is enabled, the administrator receives an email when a user is about to be banned. + +When the `auto_ban_user_on_excessive_projects_download` is not enabled, the user is not banned automatically. You can use this setup to determine the correct values of the rate limit settings. + +When both flags are enabled, the administrator receives an email when a user is about to be banned, and the user is automatically banned from the GitLab instance. + +## Configure Git abuse rate limiting + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Reporting**. +1. Expand **Git abuse rate limit**. +1. Update the Git abuse rate limit settings: + 1. Enter a number in the **Number of repositories** field, greater than or equal to `0` and less than or equal to `10,000`. This number specifies the maximum amount of unique repositories a user can download in the specified time period before they're banned. When set to `0`, Git abuse rate limiting is disabled. + 1. Enter a number in the **Reporting time period (seconds)** field, greater than or equal to `0` and less than or equal to `86,400`. This number specifies the time in seconds a user can download the maximum amount of repositories before they're banned. When set to `0`, Git abuse rate limiting is disabled. + 1. Optional. Exclude users by adding them to the **Excluded users** field. Excluded users are not automatically banned. +1. Select **Save changes**. diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index 7f37c99259a..638b61f6197 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -91,20 +91,20 @@ The value is in MB and the default is 100MB per job. To change it at the: - Instance level: 1. On the top bar, select **Menu > Admin**. - 1. On the left sidebar, select **Settings > CI/CD**. - 1. Change the value of maximum artifacts size (in MB). + 1. On the left sidebar, select **Settings > CI/CD > Continuous Integration and Deployment**. + 1. Change the value of **Maximum artifacts size (MB)**. 1. Select **Save changes** for the changes to take effect. - Group level (this overrides the instance setting): 1. Go to the group's **Settings > CI/CD > General Pipelines**. - 1. Change the value of **maximum artifacts size (in MB)**. + 1. Change the value of **Maximum artifacts size** (in MB). 1. Select **Save changes** for the changes to take effect. - Project level (this overrides the instance and group settings): 1. Go to the project's **Settings > CI/CD > General Pipelines**. - 1. Change the value of **maximum artifacts size (in MB)**. + 1. Change the value of **maximum artifacts size** (in MB). 1. Select **Save changes** for the changes to take effect. NOTE: diff --git a/doc/user/admin_area/settings/external_authorization.md b/doc/user/admin_area/settings/external_authorization.md index ef980981fec..d6e6deb0274 100644 --- a/doc/user/admin_area/settings/external_authorization.md +++ b/doc/user/admin_area/settings/external_authorization.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # External authorization control **(FREE SELF)** -> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27056) from GitLab Premium to GitLab Free in 11.10. +> [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/27056) from GitLab Premium to GitLab Free in 11.10. In highly controlled environments, it may be necessary for access policy to be controlled by an external service that permits access based on project diff --git a/doc/user/admin_area/settings/gitaly_timeouts.md b/doc/user/admin_area/settings/gitaly_timeouts.md index 42e0c9faf9f..8866a044241 100644 --- a/doc/user/admin_area/settings/gitaly_timeouts.md +++ b/doc/user/admin_area/settings/gitaly_timeouts.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Gitaly timeouts **(FREE SELF)** diff --git a/doc/user/admin_area/settings/incident_management_rate_limits.md b/doc/user/admin_area/settings/incident_management_rate_limits.md new file mode 100644 index 00000000000..ed2d707af0a --- /dev/null +++ b/doc/user/admin_area/settings/incident_management_rate_limits.md @@ -0,0 +1,38 @@ +--- +type: reference +stage: Monitor +group: Respond +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Incident management rate limits **(ULTIMATE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/17859) in GitLab 12.5. + +You can limit the number of inbound alerts for [incidents](../../../operations/incident_management/incidents.md) +that can be created in a period of time. The inbound [incident management](../../../operations/incident_management/index.md) +alert limit can help prevent overloading your incident responders by reducing the +number of alerts or duplicate issues. + +As an example, if you set a limit of `10` requests every `60` seconds, +and `11` requests are sent to an [alert integration endpoint](../../../operations/incident_management/integrations.md) within one minute, +the eleventh request is blocked. Access to the endpoint is allowed again after one minute. + +This limit is: + +- Applied independently per project. +- Not applied per IP address. +- Disabled by default. + +Requests that exceed the limit are logged into `auth.log`. + +## Set a limit on inbound alerts + +To set inbound incident management alert limits: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Network**. +1. Expand **Incident Management Limits**. +1. Select the **Enable Incident Management inbound alert limit** checkbox. +1. Optional. Input a custom value for **Maximum requests per project per rate limit period**. Default is 3600. +1. Optional. Input a custom value for **Rate limit period**. Default is 3600 seconds. diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index 034a432c570..2e27b213f16 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -162,6 +162,7 @@ The **Reporting** settings contain: - [Spam and Anti-bot Protection](../../../integration/recaptcha.md) - Enable anti-spam services, like reCAPTCHA, Akismet, or [Spamcheck](../reporting/spamcheck.md), and set IP limits. - [Abuse reports](../review_abuse_reports.md) - Set notification email for abuse reports. +- [Git abuse rate limit](../reporting/git_abuse_rate_limit.md) - Configure Git abuse rate limit settings. **(ULTIMATE SELF)** ### Repository diff --git a/doc/user/admin_area/settings/usage_statistics.md b/doc/user/admin_area/settings/usage_statistics.md index c74906c2762..65712a9a85c 100644 --- a/doc/user/admin_area/settings/usage_statistics.md +++ b/doc/user/admin_area/settings/usage_statistics.md @@ -1,5 +1,5 @@ --- -stage: Growth +stage: Analytics group: Product Intelligence info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/analytics/ci_cd_analytics.md b/doc/user/analytics/ci_cd_analytics.md index 920b651c094..f4075c3420b 100644 --- a/doc/user/analytics/ci_cd_analytics.md +++ b/doc/user/analytics/ci_cd_analytics.md @@ -78,7 +78,7 @@ To view the lead time for changes chart: ![Lead time](img/lead_time_chart_v13_11.png) -## View time to restore service chart **(PREMIUM)** +## View time to restore service chart **(ULTIMATE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/356959) in GitLab 15.1 @@ -93,3 +93,17 @@ To view the time to restore service chart: 1. Select the **Time to restore service** tab. ![Lead time](img/time_to_restore_service_charts_v15_1.png) + +## View change failure rate chart **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357072) in GitLab 15.2 + +The change failure rate chart shows information about the percentage of deployments that cause an incident in a production environment. This chart is available for groups and projects. + +Change failure rate is one of the four [DORA metrics](index.md#devops-research-and-assessment-dora-key-metrics) that DevOps teams use for measuring excellence in software delivery. + +To view the change failure rate chart: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Analytics > CI/CD Analytics**. +1. Select the **Change failure rate** tab. diff --git a/doc/user/analytics/img/time_to_restore_service_charts_v15_1.png b/doc/user/analytics/img/time_to_restore_service_charts_v15_1.png Binary files differindex 25aac385750..fbc59f72e72 100644 --- a/doc/user/analytics/img/time_to_restore_service_charts_v15_1.png +++ b/doc/user/analytics/img/time_to_restore_service_charts_v15_1.png diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index 91d9bd918b6..f699fa6d0fb 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -76,6 +76,8 @@ Deployment frequency displays in several charts: - [Project-level value stream analytics](value_stream_analytics.md) - [CI/CD analytics](ci_cd_analytics.md) +To retrieve metrics for deployment frequency, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. + ### Lead time for changes Lead time for changes measures the time to deliver a feature once it has been developed, @@ -87,6 +89,8 @@ Lead time for changes displays in several charts: - [Project-level value stream analytics](value_stream_analytics.md) - [CI/CD analytics](ci_cd_analytics.md) +To retrieve metrics for lead time for changes, use the [GraphQL](../../api/graphql/reference/index.md) or the [REST](../../api/dora/metrics.md) APIs. + ### Time to restore service Time to restore service measures how long it takes an organization to recover from a failure in production. @@ -122,12 +126,12 @@ To retrieve metrics for change failure rate, use the [GraphQL](../../api/graphql | Metric | Level | API | UI chart | Comments | |---------------------------|-------------------------|-------------------------------------|---------------------------------------|-------------------------------| -| `deployment_frequency` | Project | [GitLab 13.7 and later](../../api/dora/metrics.md) | GitLab 14.8 and later | The [previous API endpoint](../../api/dora4_project_analytics.md) was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/323713) in 13.10. | +| `deployment_frequency` | Project | [GitLab 13.7 and later](../../api/dora/metrics.md) | GitLab 14.8 and later | The previous API endpoint was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/323713) in 13.10. | | `deployment_frequency` | Group | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 13.12 and later | | | `lead_time_for_changes` | Project | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 13.11 and later | Unit in seconds. Aggregation method is median. | | `lead_time_for_changes` | Group | [GitLab 13.10 and later](../../api/dora/metrics.md) | GitLab 14.0 and later | Unit in seconds. Aggregation method is median. | -| `time_to_restore_service` | Project and group | [GitLab 14.9 and later](../../api/dora/metrics.md) | Not supported | | -| `change_failure_rate` | Project and group | [GitLab 14.10 and later](../../api/dora/metrics.md) | Not supported | | +| `time_to_restore_service` | Project and group | [GitLab 14.9 and later](../../api/dora/metrics.md) | GitLab 15.1 and later | Unit in days. Aggregation method is median. | +| `change_failure_rate` | Project and group | [GitLab 14.10 and later](../../api/dora/metrics.md) | GitLab 15.2 and later | Percentage of deployments. | | ## Definitions diff --git a/doc/user/analytics/value_stream_analytics.md b/doc/user/analytics/value_stream_analytics.md index c0f97369740..a71136628cf 100644 --- a/doc/user/analytics/value_stream_analytics.md +++ b/doc/user/analytics/value_stream_analytics.md @@ -34,7 +34,7 @@ To view value stream analytics for your project: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Analytics > Value stream**. -1. To view metrics for each stage, above the **Filter results** text box, select a stage. +1. To view metrics for a particular stage, select a stage below the **Filter results** text box. 1. Optional. Filter the results: 1. Select the **Filter results** text box. 1. Select a parameter. diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index cbe20ecde30..96236f60417 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -407,6 +407,11 @@ The API fuzzing behavior can be changed through CI/CD variables. From GitLab 13.12 and later, the default API fuzzing configuration file is `.gitlab/gitlab-api-fuzzing-config.yml`. In GitLab 14.0 and later, API fuzzing configuration files must be in your repository's `.gitlab` directory instead of your repository's root. +WARNING: +All customization of GitLab security scanning tools should be tested in a merge request before +merging these changes to the default branch. Failure to do so can give unexpected results, +including a large number of false positives. + ### Authentication Authentication is handled by providing the authentication token as a header or cookie. You can @@ -854,6 +859,9 @@ Optionally: - `FUZZAPI_PRE_SCRIPT`: Script to install runtimes or dependencies before the analyzer starts. +WARNING: +To execute scripts in Alpine Linux you must first use the command [`chmod`](https://www.gnu.org/software/coreutils/manual/html_node/chmod-invocation.html) to set the [execution permission](https://www.gnu.org/software/coreutils/manual/html_node/Setting-Permissions.html). For example, to set the execution permission of `script.py` for everyone, use the command: `chmod a+x script.py`. If needed, you can version your `script.py` with the execution permission already set. + ```yaml stages: - fuzz @@ -902,7 +910,9 @@ import requests import backoff # [1] Store log file in directory indicated by env var CI_PROJECT_DIR -working_directory = os.environ['CI_PROJECT_DIR'] +working_directory = os.environ.get( 'CI_PROJECT_DIR') +overrides_file_name = os.environ.get('FUZZAPI_OVERRIDES_FILE', 'api-fuzzing-overrides.json') +overrides_file_path = os.path.join(working_directory, overrides_file_name) # [2] File name should match the pattern: gl-*.log log_file_path = os.path.join(working_directory, 'gl-user-overrides.log') @@ -916,8 +926,11 @@ logging.basicConfig(filename=log_file_path, level=logging.DEBUG) requests.exceptions.ConnectionError), max_time=30) def get_auth_response(): - return requests.get('https://authorization.service/api/get_api_token', auth=(os.environ['AUTH_USER'], os.environ['AUTH_PWD'])) - + authorization_url = 'https://authorization.service/api/get_api_token' + return requests.get( + f'{authorization_url}', + auth=(os.environ.get('AUTH_USER'), os.environ.get('AUTH_PWD')) + ) # In our example, access token is retrieved from a given endpoint try: @@ -939,14 +952,14 @@ try: # requests.ReadTimeout : The server did not send any data in the allotted amount of time. # requests.TooManyRedirects : The request exceeds the configured number of maximum redirections # requests.exceptions.RequestException : All exceptions that related to Requests +except json.JSONDecodeError as json_decode_error: + # logs errors related decoding JSON response + logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}') + raise except requests.exceptions.RequestException as requests_error: # logs exceptions related to `Requests` logging.error(f'Error, failed while performing HTTP request. Error message: {requests_error}') raise -except requests.exceptions.JSONDecodeError as json_decode_error: - # logs errors related decoding JSON response - logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}') - raise except Exception as e: # logs any other error logging.error(f'Error, unknown error while retrieving access token. Error message: {e}') @@ -961,8 +974,6 @@ overrides_data = { } # log entry informing about the file override computation -overrides_file_path = os.path.join( - working_directory, "api-fuzzing-overrides.json") logging.info("Creating overrides file: %s" % overrides_file_path) # attempts to overwrite the file @@ -975,7 +986,7 @@ try: fd.write(json.dumps(overrides_data).encode('utf-8')) except Exception as e: # logs any other error - logging.error(f'Error, unkown error when overwritng file {overrides_file_path}. Error message: {e}') + logging.error(f'Error, unknown error when overwriting file {overrides_file_path}. Error message: {e}') raise # logs informing override has finished successfully @@ -998,6 +1009,7 @@ echo "**** install python dependencies ****" python3 -m ensurepip pip3 install --no-cache --upgrade \ pip \ + requests \ backoff echo "**** python dependencies installed ****" @@ -1028,7 +1040,7 @@ In the previous sample, you could use the script `user-pre-scan-set-up.sh` to al ### Exclude Paths -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211892) in GitLab 14.0. When testing an API it can be useful to exclude certain paths. For example, you might exclude testing of an authentication service or an older version of the API. To exclude paths, use the `FUZZAPI_EXCLUDE_PATHS` CI/CD variable . This variable is specified in your `.gitlab-ci.yml` file. To exclude multiple paths, separate entries using the `;` character. In the provided paths you can use a single character wildcard `?` and `*` for a multiple character wildcard. diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index 09292dcb92b..9ca1a6f125f 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -53,7 +53,7 @@ You can configure the following security controls: enable Container Scanning. For more details, see [Enable Container Scanning through an automatic merge request](../container_scanning/index.md#enable-container-scanning-through-an-automatic-merge-request). - [Operational Container Scanning](../../clusters/agent/vulnerabilities.md) - - Can be configured by adding a configuration block to your agent configuration. For more details, read [Operational Container Scanning](../../clusters/agent/vulnerabilities.md#enable-cluster-vulnerability-scanning). + - Can be configured by adding a configuration block to your agent configuration. For more details, read [Operational Container Scanning](../../clusters/agent/vulnerabilities.md#enable-operational-container-scanning). - [Secret Detection](../secret_detection/index.md) - Select **Configure with a merge request** to create a merge request with the changes required to enable Secret Detection. For more details, read [Enable Secret Detection via an automatic merge request](../secret_detection/index.md#enable-secret-detection-via-an-automatic-merge-request). diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index c41385a3569..cf864068e44 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -178,6 +178,8 @@ include: DOCKER_PASSWORD: "$AWS_ECR_PASSWORD" ``` +Authenticating to a remote registry is not supported when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) is enabled. + #### Dependency list > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345434) in GitLab 14.6. @@ -231,7 +233,12 @@ between GitLab Dependency Scanning and Container Scanning for more details on wh #### Available CI/CD variables -You can [configure](#customizing-the-container-scanning-settings) analyzers by using the following CI/CD variables: +You can [configure](#customizing-the-container-scanning-settings) analyzers by using the following CI/CD variables. + +WARNING: +All customization of GitLab security scanning tools should be tested in a merge request before +merging these changes to the default branch. Failure to do so can give unexpected results, +including a large number of false positives. | CI/CD Variable | Default | Description | Scanner | | ------------------------------ | ------------- | ----------- | ------------ | @@ -248,8 +255,8 @@ You can [configure](#customizing-the-container-scanning-settings) analyzers by u | `CS_REGISTRY_INSECURE` | `"false"` | Allow access to insecure registries (HTTP only). Should only be set to `true` when testing the image locally. Works with all scanners, but the registry must listen on port `80/tcp` for Trivy to work. | All | | `CS_SEVERITY_THRESHOLD` | `UNKNOWN` | Severity level threshold. The scanner outputs vulnerabilities with severity level higher than or equal to this threshold. Supported levels are Unknown, Low, Medium, High, and Critical. | Trivy | | `DOCKER_IMAGE` | `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG` | The Docker image to be scanned. If set, this variable overrides the `$CI_APPLICATION_REPOSITORY` and `$CI_APPLICATION_TAG` variables. | All | -| `DOCKER_PASSWORD` | `$CI_REGISTRY_PASSWORD` | Password for accessing a Docker registry requiring authentication. The default is only set if `$DOCKER_IMAGE` resides at [`$CI_REGISTRY`](../../../ci/variables/predefined_variables.md). | All | -| `DOCKER_USER` | `$CI_REGISTRY_USER` | Username for accessing a Docker registry requiring authentication. The default is only set if `$DOCKER_IMAGE` resides at [`$CI_REGISTRY`](../../../ci/variables/predefined_variables.md). | All | +| `DOCKER_PASSWORD` | `$CI_REGISTRY_PASSWORD` | Password for accessing a Docker registry requiring authentication. The default is only set if `$DOCKER_IMAGE` resides at [`$CI_REGISTRY`](../../../ci/variables/predefined_variables.md). Not supported when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) is enabled. | All | +| `DOCKER_USER` | `$CI_REGISTRY_USER` | Username for accessing a Docker registry requiring authentication. The default is only set if `$DOCKER_IMAGE` resides at [`$CI_REGISTRY`](../../../ci/variables/predefined_variables.md). Not supported when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) is enabled. | All | | `DOCKERFILE_PATH` | `Dockerfile` | The path to the `Dockerfile` to use for generating remediations. By default, the scanner looks for a file named `Dockerfile` in the root directory of the project. You should configure this variable only if your `Dockerfile` is in a non-standard location, such as a subdirectory. See [Solutions for vulnerabilities](#solutions-for-vulnerabilities-auto-remediation) for more details. | All | | `SECURE_LOG_LEVEL` | `info` | Set the minimum logging level. Messages of this logging level or higher are output. From highest to lowest severity, the logging levels are: `fatal`, `error`, `warn`, `info`, `debug`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10880) in GitLab 13.1. | All | @@ -296,6 +303,10 @@ support `-fips`. Starting with GitLab 14.10, `-fips` is automatically added to `CS_ANALYZER_IMAGE` when FIPS mode is enabled in the GitLab instance. +Container scanning of images in authenticated registries is not supported when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) +is enabled. When `CI_GITLAB_FIPS_MODE` is `"true"`, and `DOCKER_USER` or `DOCKER_PASSWORD` is set, +the analyzer exits with an error and does not perform the scan. + ### Enable Container Scanning through an automatic merge request > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6334) in GitLab 14.9. @@ -661,6 +672,8 @@ Also: - Consider creating credentials with read-only permissions and rotating them regularly if the options aren't selected. +Scanning images in external private registries is not supported when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) is enabled. + ## Running the standalone container scanning tool It's possible to run the [GitLab container scanning tool](https://gitlab.com/gitlab-org/security-products/analyzers/container-scanning) diff --git a/doc/user/application_security/coverage_fuzzing/index.md b/doc/user/application_security/coverage_fuzzing/index.md index b2b7dd85468..ac3b266ad48 100644 --- a/doc/user/application_security/coverage_fuzzing/index.md +++ b/doc/user/application_security/coverage_fuzzing/index.md @@ -113,6 +113,11 @@ job. If you include these keys in your own job, you must copy their original con Use the following variables to configure coverage-guided fuzz testing in your CI/CD pipeline. +WARNING: +All customization of GitLab security scanning tools should be tested in a merge request before +merging these changes to the default branch. Failure to do so can give unexpected results, including +a large number of false positives. + | CI/CD variable | Description | |---------------------------|---------------------------------------------------------------------------------| | `COVFUZZ_ADDITIONAL_ARGS` | Arguments passed to `gitlab-cov-fuzz`. Used to customize the behavior of the underlying fuzzing engine. Read the fuzzing engine's documentation for a complete list of arguments. | diff --git a/doc/user/application_security/dast/checks/1004.1.md b/doc/user/application_security/dast/checks/1004.1.md index 72af1156b95..40139f2aa8a 100644 --- a/doc/user/application_security/dast/checks/1004.1.md +++ b/doc/user/application_security/dast/checks/1004.1.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Description -The {cookie_name} cookie was transmitted in a `Set-Cookie` header without the `HttpOnly` attribute set. +The cookie was transmitted in a `Set-Cookie` header without the `HttpOnly` attribute set. To prevent JavaScript being able to access the cookie value - usually via `document.cookies` - all cookies that are used for authorization should have the `HttpOnly` attribute set. diff --git a/doc/user/application_security/dast/checks/16.10.md b/doc/user/application_security/dast/checks/16.10.md new file mode 100644 index 00000000000..67368d80022 --- /dev/null +++ b/doc/user/application_security/dast/checks/16.10.md @@ -0,0 +1,30 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Content-Security-Policy violations + +## Description + +A `Content-Security-Policy` (CSP) was identified on the target site that is reporting violations when +attempting to load the page in a browser. This may cause disruption to your users when attempting to visit the page. + +## Remediation + +Review the violations to determine if any action is necessary. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 16.10 | true | 16 | Passive | Info | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/16.html) +- [OWASP](https://cheatsheetseries.owasp.org/cheatsheets/Content_Security_Policy_Cheat_Sheet.html) +- [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) +- [Content Security Policy Level 3](https://www.w3.org/TR/CSP3/) +- [CSP Evaluator](https://csp-evaluator.withgoogle.com/) diff --git a/doc/user/application_security/dast/checks/16.3.md b/doc/user/application_security/dast/checks/16.3.md index 6f80a2a32c6..e4fc2468dae 100644 --- a/doc/user/application_security/dast/checks/16.3.md +++ b/doc/user/application_security/dast/checks/16.3.md @@ -32,4 +32,4 @@ information from the `X-Powered-By` header. ## Links - [CWE](https://cwe.mitre.org/data/definitions/16.html) -- [PHP `expose_php`](https://www.php.net/manual/en/ini.core.php#ini.expose-php) +- [PHP expose_php](https://www.php.net/manual/en/ini.core.php#ini.expose-php) diff --git a/doc/user/application_security/dast/checks/16.5.md b/doc/user/application_security/dast/checks/16.5.md index e03da3043ef..28bb9f7ee4b 100644 --- a/doc/user/application_security/dast/checks/16.5.md +++ b/doc/user/application_security/dast/checks/16.5.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Description -The target website returns AspNet headers and version information of this website. By +The target website returns AspNet header(s) and version information of this website. By exposing these values attackers may attempt to identify if the target software is vulnerable to known vulnerabilities, or catalog known sites running particular versions to exploit in the future when a vulnerability is identified in the particular version. diff --git a/doc/user/application_security/dast/checks/16.6.md b/doc/user/application_security/dast/checks/16.6.md index 9cbcde669a0..ddd3a10c5f8 100644 --- a/doc/user/application_security/dast/checks/16.6.md +++ b/doc/user/application_security/dast/checks/16.6.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Description -The target website returns AspNet headers along with version information of this website. By +The target website returns AspNet header(s) along with version information of this website. By exposing these values attackers may attempt to identify if the target software is vulnerable to known vulnerabilities. Or catalog known sites running particular versions to exploit in the future when a vulnerability is identified in the particular version. diff --git a/doc/user/application_security/dast/checks/16.8.md b/doc/user/application_security/dast/checks/16.8.md new file mode 100644 index 00000000000..c9beba4544e --- /dev/null +++ b/doc/user/application_security/dast/checks/16.8.md @@ -0,0 +1,30 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Content-Security-Policy analysis + +## Description + +A `Content-Security-Policy` (CSP) was identified on the target site. CSP can aid in hardening +a website against various client side attacks such as Cross-Site Scripting (XSS). + +## Remediation + +Follow the recommendations to determine if any actions are necessary to harden this `Content-Security-Policy`. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 16.8 | true | 16 | Passive | Info | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/16.html) +- [OWASP](https://cheatsheetseries.owasp.org/cheatsheets/Content_Security_Policy_Cheat_Sheet.html) +- [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) +- [Content Security Policy Level 3](https://www.w3.org/TR/CSP3/) +- [CSP Evaluator](https://csp-evaluator.withgoogle.com/) diff --git a/doc/user/application_security/dast/checks/16.9.md b/doc/user/application_security/dast/checks/16.9.md new file mode 100644 index 00000000000..c3e4431e415 --- /dev/null +++ b/doc/user/application_security/dast/checks/16.9.md @@ -0,0 +1,32 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Content-Security-Policy-Report-Only analysis + +## Description + +A `Content-Security-Policy-Report-Only` (CSPRO) was identified on the target site. CSP-Report-Only headers +aid in determining how to implement a `Content-Security-Policy` that does not disrupt normal use of the target +site. + +## Remediation + +Follow the recommendations to determine if any actions are necessary to harden this `Content-Security-Policy-Report-Only`. +After all alerts have been resolved, we recommend that this header be changed to `Content-Security-Policy`. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 16.9 | true | 16 | Passive | Info | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/16.html) +- [OWASP](https://cheatsheetseries.owasp.org/cheatsheets/Content_Security_Policy_Cheat_Sheet.html) +- [MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP) +- [Content Security Policy Level 3](https://www.w3.org/TR/CSP3/) +- [CSP Evaluator](https://csp-evaluator.withgoogle.com/) diff --git a/doc/user/application_security/dast/checks/209.2.md b/doc/user/application_security/dast/checks/209.2.md new file mode 100644 index 00000000000..2060bb1802b --- /dev/null +++ b/doc/user/application_security/dast/checks/209.2.md @@ -0,0 +1,43 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Generation of database error message containing sensitive information + +## Description + +The application was found to return database error messages. Determining the type of database may assist attackers in exploiting +SQL Injection attacks against the system. While debug messages are helpful during development and debugging, they should not be +presented to users when an error occurs. + +## Remediation + +Applications should handle database error conditions internally and map known failure types to error codes that can be displayed +to a user. These error codes should be customized to the application and returned along with the relevant HTTP error code. + +When an error occurs, the application identifies the error type or class, and displays a numerical value to the +user. Requests should also be tracked so when a user is presented with an error code, it has a corresponding request ID. +Support teams can then correlate the HTTP error, the customized error code, and the request ID in the log files to +determine the root cause of the error without leaking details to the end user. + +Example of returning customized errors: + +```plaintext +HTTP/1.1 500 Internal Server Error +... +Error [0004] Occurred, please contact support or re-try your request again shortly. +Request ID [a4bc91def12] +... +``` + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 209.2 | false | 209 | Passive | Low | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/209.html) diff --git a/doc/user/application_security/dast/checks/287.1.md b/doc/user/application_security/dast/checks/287.1.md new file mode 100644 index 00000000000..06b7e7b4b2e --- /dev/null +++ b/doc/user/application_security/dast/checks/287.1.md @@ -0,0 +1,33 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Insecure authentication over HTTP (Basic Authentication) + +## Description + +The target application was found to authenticate users using the Basic Authentication scheme over HTTP. +Basic Authentication base64 encodes the username and password and sends it in the `Authentication` header. +Attackers who are in between the communication path (or on the same local network) of the client and server +could use packet sniffers to read and decode the username and password. + +## Remediation + +If possible, switch to a more robust method to authenticate users such as OAuth 2.0, or storing usernames +and passwords in a data store protected by the Argon2id algorithm. If Basic Authentication must be used, +ensure credentials are only transmitted over secure channels such as HTTPS/TLS. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 287.1 | false | 287 | Passive | Medium | + +## Links + +- [OWASP](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html) +- [OAuth 2.0](https://oauth.net/2/) +- [CWE-287](https://cwe.mitre.org/data/definitions/287.html) +- [RFC](https://datatracker.ietf.org/doc/html/rfc7617) diff --git a/doc/user/application_security/dast/checks/287.2.md b/doc/user/application_security/dast/checks/287.2.md new file mode 100644 index 00000000000..2215b72f47a --- /dev/null +++ b/doc/user/application_security/dast/checks/287.2.md @@ -0,0 +1,35 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Insecure authentication over HTTP (Digest Authentication) + +## Description + +The target application was found to authenticate users using the Digest Authentication scheme over HTTP. +Digest Authentication uses an insecure hashing algorithm (MD5) to hash the username and password and sends +it in the `Authentication` header. Attackers who are in between the communication path (or on the same +local network) of the client and server could use packet sniffers to modify the server's response parameters +to downgrade the security of the digest access authentication mode. Additionally, the server stores the +hashed credentials, usually in a file called `.htpasswd`. Tools are readily available to crack these passwords. + +## Remediation + +If possible, switch to a more robust method to authenticate users such as OAuth 2.0, or storing usernames +and passwords in a data store protected by the Argon2id algorithm. If Digest Authentication must be used, +ensure credentials are only transmitted over secure channels such as HTTPS/TLS. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 287.2 | false | 287 | Passive | Low | + +## Links + +- [OWASP](https://cheatsheetseries.owasp.org/cheatsheets/Password_Storage_Cheat_Sheet.html) +- [OAuth 2.0](https://oauth.net/2/) +- [CWE-287](https://cwe.mitre.org/data/definitions/287.html) +- [RFC](https://datatracker.ietf.org/doc/html/rfc2069) diff --git a/doc/user/application_security/dast/checks/601.1.md b/doc/user/application_security/dast/checks/601.1.md index 26ccd877104..60249c2562d 100644 --- a/doc/user/application_security/dast/checks/601.1.md +++ b/doc/user/application_security/dast/checks/601.1.md @@ -8,17 +8,17 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Description -This site was found to allow open redirects from user supplied input. Open redirects are commonly -abused in phishing attacks where the original domain or URL looks like a legitimate link, but then -redirects a user to a malicious site. An example would be -`https://example.com/redirect?url=https://%62%61%64%2e%63%6f%6d%2f%66%61%6b%65%6c%6f%67%69%6e` which, +This site was found to allow open redirects from user supplied input. Open redirects are commonly +abused in phishing attacks where the original domain or URL looks like a legitimate link, but then +redirects a user to a malicious site. An example would be +`https://example.com/redirect?url=https://%62%61%64%2e%63%6f%6d%2f%66%61%6b%65%6c%6f%67%69%6e` which, when decoded turns into `bad.com/fakelogin`. ## Remediation -Never redirect a client based on user input found in a `GET` request. It is recommended that the list -of target links to redirect a user to are contained server side, and retrieved using a numerical value -as an index to return the link to be redirected to. For example, `/redirect?id=1` would cause the +Never redirect a client based on user input found in a `GET` request. It is recommended that the list +of target links to redirect a user to are contained server side, and retrieved using a numerical value +as an index to return the link to be redirected to. For example, `/redirect?id=1` would cause the application to look up the `1` index and return a URL such as `https://example.com`. This URL would then be used to redirect the user, using the 301 response code and `Location` header. diff --git a/doc/user/application_security/dast/checks/614.1.md b/doc/user/application_security/dast/checks/614.1.md index ec68ce33529..d5c7476716f 100644 --- a/doc/user/application_security/dast/checks/614.1.md +++ b/doc/user/application_security/dast/checks/614.1.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Description -The {cookie_name} cookie was transmitted in a `Set-Cookie` response without the `Secure` attribute set. +The cookie was transmitted in a `Set-Cookie` response without the `Secure` attribute set. To prevent sensitive cookie values being accidentally transmitted over clear-text HTTP we recommended that cookies are declared with the `Secure` attribute. diff --git a/doc/user/application_security/dast/checks/798.1.md b/doc/user/application_security/dast/checks/798.1.md new file mode 100644 index 00000000000..819ae92cfdc --- /dev/null +++ b/doc/user/application_security/dast/checks/798.1.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Adafruit API Key + +## Description + +The response body contains content that matches the pattern of a Adafruit API Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.1 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.10.md b/doc/user/application_security/dast/checks/798.10.md new file mode 100644 index 00000000000..14723c81f17 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.10.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Asana Client Secret + +## Description + +The response body contains content that matches the pattern of a Asana Client Secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.10 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.100.md b/doc/user/application_security/dast/checks/798.100.md new file mode 100644 index 00000000000..07bd24211c7 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.100.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Sendbird Access Token + +## Description + +The response body contains content that matches the pattern of a Sendbird Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.100 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.101.md b/doc/user/application_security/dast/checks/798.101.md new file mode 100644 index 00000000000..ea102147100 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.101.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token SendGrid API token + +## Description + +The response body contains content that matches the pattern of a SendGrid API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.101 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.102.md b/doc/user/application_security/dast/checks/798.102.md new file mode 100644 index 00000000000..8a40475190a --- /dev/null +++ b/doc/user/application_security/dast/checks/798.102.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Sendinblue API token + +## Description + +The response body contains content that matches the pattern of a Sendinblue API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.102 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.103.md b/doc/user/application_security/dast/checks/798.103.md new file mode 100644 index 00000000000..3d91f7f3b80 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.103.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Sentry Access Token + +## Description + +The response body contains content that matches the pattern of a Sentry Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.103 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.104.md b/doc/user/application_security/dast/checks/798.104.md new file mode 100644 index 00000000000..316998615ff --- /dev/null +++ b/doc/user/application_security/dast/checks/798.104.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Shippo API token + +## Description + +The response body contains content that matches the pattern of a Shippo API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.104 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.105.md b/doc/user/application_security/dast/checks/798.105.md new file mode 100644 index 00000000000..20618a9d555 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.105.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Shopify access token + +## Description + +The response body contains content that matches the pattern of a Shopify access token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.105 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.106.md b/doc/user/application_security/dast/checks/798.106.md new file mode 100644 index 00000000000..4f552302e85 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.106.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Shopify custom access token + +## Description + +The response body contains content that matches the pattern of a Shopify custom access token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.106 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.107.md b/doc/user/application_security/dast/checks/798.107.md new file mode 100644 index 00000000000..2a5961b3905 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.107.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Shopify private app access token + +## Description + +The response body contains content that matches the pattern of a Shopify private app access token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.107 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.108.md b/doc/user/application_security/dast/checks/798.108.md new file mode 100644 index 00000000000..23968bcf660 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.108.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Shopify shared secret + +## Description + +The response body contains content that matches the pattern of a Shopify shared secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.108 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.109.md b/doc/user/application_security/dast/checks/798.109.md new file mode 100644 index 00000000000..57d6823d8a9 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.109.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Slack token + +## Description + +The response body contains content that matches the pattern of a Slack token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.109 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.11.md b/doc/user/application_security/dast/checks/798.11.md new file mode 100644 index 00000000000..b12f86ba800 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.11.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Atlassian API token + +## Description + +The response body contains content that matches the pattern of a Atlassian API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.11 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.110.md b/doc/user/application_security/dast/checks/798.110.md new file mode 100644 index 00000000000..8ac7a8a4be2 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.110.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Slack Webhook + +## Description + +The response body contains content that matches the pattern of a Slack Webhook. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.110 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.111.md b/doc/user/application_security/dast/checks/798.111.md new file mode 100644 index 00000000000..ff05dcfe55b --- /dev/null +++ b/doc/user/application_security/dast/checks/798.111.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Stripe + +## Description + +The response body contains content that matches the pattern of a Stripe. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.111 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.112.md b/doc/user/application_security/dast/checks/798.112.md new file mode 100644 index 00000000000..4f5f89dab9c --- /dev/null +++ b/doc/user/application_security/dast/checks/798.112.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Square Access Token + +## Description + +The response body contains content that matches the pattern of a Square Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.112 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.113.md b/doc/user/application_security/dast/checks/798.113.md new file mode 100644 index 00000000000..3f8d1a88ec0 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.113.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Squarespace Access Token + +## Description + +The response body contains content that matches the pattern of a Squarespace Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.113 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.114.md b/doc/user/application_security/dast/checks/798.114.md new file mode 100644 index 00000000000..0b8235af8c7 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.114.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token SumoLogic Access ID + +## Description + +The response body contains content that matches the pattern of a SumoLogic Access ID. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.114 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.115.md b/doc/user/application_security/dast/checks/798.115.md new file mode 100644 index 00000000000..052502ea962 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.115.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token SumoLogic Access Token + +## Description + +The response body contains content that matches the pattern of a SumoLogic Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.115 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.116.md b/doc/user/application_security/dast/checks/798.116.md new file mode 100644 index 00000000000..7b1f0eb907d --- /dev/null +++ b/doc/user/application_security/dast/checks/798.116.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Travis CI Access Token + +## Description + +The response body contains content that matches the pattern of a Travis CI Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.116 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.117.md b/doc/user/application_security/dast/checks/798.117.md new file mode 100644 index 00000000000..5cd9817795a --- /dev/null +++ b/doc/user/application_security/dast/checks/798.117.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Twilio API Key + +## Description + +The response body contains content that matches the pattern of a Twilio API Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.117 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.118.md b/doc/user/application_security/dast/checks/798.118.md new file mode 100644 index 00000000000..a74233429df --- /dev/null +++ b/doc/user/application_security/dast/checks/798.118.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Twitch API token + +## Description + +The response body contains content that matches the pattern of a Twitch API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.118 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.119.md b/doc/user/application_security/dast/checks/798.119.md new file mode 100644 index 00000000000..80fada87b1c --- /dev/null +++ b/doc/user/application_security/dast/checks/798.119.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Twitter API Key + +## Description + +The response body contains content that matches the pattern of a Twitter API Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.119 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.12.md b/doc/user/application_security/dast/checks/798.12.md new file mode 100644 index 00000000000..6f8d0c83a94 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.12.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token AWS + +## Description + +The response body contains content that matches the pattern of a AWS. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.12 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.120.md b/doc/user/application_security/dast/checks/798.120.md new file mode 100644 index 00000000000..639b5c6ffc2 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.120.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Twitter API Secret + +## Description + +The response body contains content that matches the pattern of a Twitter API Secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.120 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.121.md b/doc/user/application_security/dast/checks/798.121.md new file mode 100644 index 00000000000..e574760baa2 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.121.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Twitter Access Token + +## Description + +The response body contains content that matches the pattern of a Twitter Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.121 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.122.md b/doc/user/application_security/dast/checks/798.122.md new file mode 100644 index 00000000000..9acb82a6062 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.122.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Twitter Access Secret + +## Description + +The response body contains content that matches the pattern of a Twitter Access Secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.122 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.123.md b/doc/user/application_security/dast/checks/798.123.md new file mode 100644 index 00000000000..5d5c9df5f40 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.123.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Twitter Bearer Token + +## Description + +The response body contains content that matches the pattern of a Twitter Bearer Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.123 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.124.md b/doc/user/application_security/dast/checks/798.124.md new file mode 100644 index 00000000000..4900ca44ba4 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.124.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Typeform API token + +## Description + +The response body contains content that matches the pattern of a Typeform API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.124 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.125.md b/doc/user/application_security/dast/checks/798.125.md new file mode 100644 index 00000000000..1111ef91491 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.125.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Yandex API Key + +## Description + +The response body contains content that matches the pattern of a Yandex API Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.125 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.126.md b/doc/user/application_security/dast/checks/798.126.md new file mode 100644 index 00000000000..6253f9a4a92 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.126.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Yandex AWS Access Token + +## Description + +The response body contains content that matches the pattern of a Yandex AWS Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.126 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.127.md b/doc/user/application_security/dast/checks/798.127.md new file mode 100644 index 00000000000..86bb9613f16 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.127.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Yandex Access Token + +## Description + +The response body contains content that matches the pattern of a Yandex Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.127 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.128.md b/doc/user/application_security/dast/checks/798.128.md new file mode 100644 index 00000000000..0db8cdd8005 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.128.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Zendesk Secret Key + +## Description + +The response body contains content that matches the pattern of a Zendesk Secret Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.128 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.13.md b/doc/user/application_security/dast/checks/798.13.md new file mode 100644 index 00000000000..8cf2f7c2895 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.13.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Bitbucket Client ID + +## Description + +The response body contains content that matches the pattern of a Bitbucket Client ID. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.13 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.14.md b/doc/user/application_security/dast/checks/798.14.md new file mode 100644 index 00000000000..85b88660b5a --- /dev/null +++ b/doc/user/application_security/dast/checks/798.14.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Bitbucket Client Secret + +## Description + +The response body contains content that matches the pattern of a Bitbucket Client Secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.14 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.15.md b/doc/user/application_security/dast/checks/798.15.md new file mode 100644 index 00000000000..51f2fae0021 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.15.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Bittrex Access Key + +## Description + +The response body contains content that matches the pattern of a Bittrex Access Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.15 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.16.md b/doc/user/application_security/dast/checks/798.16.md new file mode 100644 index 00000000000..872a97e70ea --- /dev/null +++ b/doc/user/application_security/dast/checks/798.16.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Bittrex Secret Key + +## Description + +The response body contains content that matches the pattern of a Bittrex Secret Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.16 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.17.md b/doc/user/application_security/dast/checks/798.17.md new file mode 100644 index 00000000000..9e11af3bfe8 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.17.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Beamer API token + +## Description + +The response body contains content that matches the pattern of a Beamer API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.17 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.18.md b/doc/user/application_security/dast/checks/798.18.md new file mode 100644 index 00000000000..71caa0a53ba --- /dev/null +++ b/doc/user/application_security/dast/checks/798.18.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Codecov Access Token + +## Description + +The response body contains content that matches the pattern of a Codecov Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.18 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.19.md b/doc/user/application_security/dast/checks/798.19.md new file mode 100644 index 00000000000..6cfbab0e9d1 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.19.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Coinbase Access Token + +## Description + +The response body contains content that matches the pattern of a Coinbase Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.19 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.2.md b/doc/user/application_security/dast/checks/798.2.md new file mode 100644 index 00000000000..766f4c75973 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.2.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Adobe Client ID (OAuth Web) + +## Description + +The response body contains content that matches the pattern of a Adobe Client ID (OAuth Web). +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.2 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.20.md b/doc/user/application_security/dast/checks/798.20.md new file mode 100644 index 00000000000..83651142912 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.20.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Clojars API token + +## Description + +The response body contains content that matches the pattern of a Clojars API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.20 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.21.md b/doc/user/application_security/dast/checks/798.21.md new file mode 100644 index 00000000000..93bf588c84b --- /dev/null +++ b/doc/user/application_security/dast/checks/798.21.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Confluent Access Token + +## Description + +The response body contains content that matches the pattern of a Confluent Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.21 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.22.md b/doc/user/application_security/dast/checks/798.22.md new file mode 100644 index 00000000000..7a8abbce7ba --- /dev/null +++ b/doc/user/application_security/dast/checks/798.22.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Confluent Secret Key + +## Description + +The response body contains content that matches the pattern of a Confluent Secret Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.22 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.23.md b/doc/user/application_security/dast/checks/798.23.md new file mode 100644 index 00000000000..f5460e98079 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.23.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Contentful delivery API token + +## Description + +The response body contains content that matches the pattern of a Contentful delivery API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.23 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.24.md b/doc/user/application_security/dast/checks/798.24.md new file mode 100644 index 00000000000..7a01197a6b8 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.24.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Databricks API token + +## Description + +The response body contains content that matches the pattern of a Databricks API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.24 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.25.md b/doc/user/application_security/dast/checks/798.25.md new file mode 100644 index 00000000000..c5dcee20f61 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.25.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Datadog Access Token + +## Description + +The response body contains content that matches the pattern of a Datadog Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.25 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.26.md b/doc/user/application_security/dast/checks/798.26.md new file mode 100644 index 00000000000..bfa5cb0588e --- /dev/null +++ b/doc/user/application_security/dast/checks/798.26.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Discord API key + +## Description + +The response body contains content that matches the pattern of a Discord API key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.26 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.27.md b/doc/user/application_security/dast/checks/798.27.md new file mode 100644 index 00000000000..1210d91e741 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.27.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Discord client ID + +## Description + +The response body contains content that matches the pattern of a Discord client ID. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.27 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.28.md b/doc/user/application_security/dast/checks/798.28.md new file mode 100644 index 00000000000..5f4718d8eb7 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.28.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Discord client secret + +## Description + +The response body contains content that matches the pattern of a Discord client secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.28 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.29.md b/doc/user/application_security/dast/checks/798.29.md new file mode 100644 index 00000000000..90371a157a0 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.29.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Doppler API token + +## Description + +The response body contains content that matches the pattern of a Doppler API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.29 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.3.md b/doc/user/application_security/dast/checks/798.3.md new file mode 100644 index 00000000000..43d69b77337 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.3.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Adobe Client Secret + +## Description + +The response body contains content that matches the pattern of a Adobe Client Secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.3 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.30.md b/doc/user/application_security/dast/checks/798.30.md new file mode 100644 index 00000000000..db62b30b84b --- /dev/null +++ b/doc/user/application_security/dast/checks/798.30.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Dropbox API secret + +## Description + +The response body contains content that matches the pattern of a Dropbox API secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.30 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.31.md b/doc/user/application_security/dast/checks/798.31.md new file mode 100644 index 00000000000..8f03ba780e4 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.31.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Dropbox long lived API token + +## Description + +The response body contains content that matches the pattern of a Dropbox long lived API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.31 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.32.md b/doc/user/application_security/dast/checks/798.32.md new file mode 100644 index 00000000000..d2ed4af9177 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.32.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Dropbox short lived API token + +## Description + +The response body contains content that matches the pattern of a Dropbox short lived API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.32 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.33.md b/doc/user/application_security/dast/checks/798.33.md new file mode 100644 index 00000000000..5a264cf4286 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.33.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Droneci Access Token + +## Description + +The response body contains content that matches the pattern of a Droneci Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.33 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.34.md b/doc/user/application_security/dast/checks/798.34.md new file mode 100644 index 00000000000..a9b02b75230 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.34.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Duffel API token + +## Description + +The response body contains content that matches the pattern of a Duffel API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.34 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.35.md b/doc/user/application_security/dast/checks/798.35.md new file mode 100644 index 00000000000..5d35baec9bb --- /dev/null +++ b/doc/user/application_security/dast/checks/798.35.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Dynatrace API token + +## Description + +The response body contains content that matches the pattern of a Dynatrace API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.35 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.36.md b/doc/user/application_security/dast/checks/798.36.md new file mode 100644 index 00000000000..e2e0f10f842 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.36.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token EasyPost API token + +## Description + +The response body contains content that matches the pattern of a EasyPost API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.36 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.37.md b/doc/user/application_security/dast/checks/798.37.md new file mode 100644 index 00000000000..089dc8b3ecc --- /dev/null +++ b/doc/user/application_security/dast/checks/798.37.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token EasyPost test API token + +## Description + +The response body contains content that matches the pattern of a EasyPost test API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.37 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.38.md b/doc/user/application_security/dast/checks/798.38.md new file mode 100644 index 00000000000..886cfcc701b --- /dev/null +++ b/doc/user/application_security/dast/checks/798.38.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Etsy Access Token + +## Description + +The response body contains content that matches the pattern of a Etsy Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.38 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.39.md b/doc/user/application_security/dast/checks/798.39.md new file mode 100644 index 00000000000..78a66d15b89 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.39.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Facebook + +## Description + +The response body contains content that matches the pattern of a Facebook. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.39 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.4.md b/doc/user/application_security/dast/checks/798.4.md new file mode 100644 index 00000000000..2ff5db46d83 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.4.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Age secret key + +## Description + +The response body contains content that matches the pattern of a Age secret key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.4 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.40.md b/doc/user/application_security/dast/checks/798.40.md new file mode 100644 index 00000000000..e6691bb7b3a --- /dev/null +++ b/doc/user/application_security/dast/checks/798.40.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Fastly API key + +## Description + +The response body contains content that matches the pattern of a Fastly API key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.40 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.41.md b/doc/user/application_security/dast/checks/798.41.md new file mode 100644 index 00000000000..b4d097a9014 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.41.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Finicity Client Secret + +## Description + +The response body contains content that matches the pattern of a Finicity Client Secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.41 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.42.md b/doc/user/application_security/dast/checks/798.42.md new file mode 100644 index 00000000000..30c380d13a5 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.42.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Finicity API token + +## Description + +The response body contains content that matches the pattern of a Finicity API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.42 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.43.md b/doc/user/application_security/dast/checks/798.43.md new file mode 100644 index 00000000000..be984f7119a --- /dev/null +++ b/doc/user/application_security/dast/checks/798.43.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Flickr Access Token + +## Description + +The response body contains content that matches the pattern of a Flickr Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.43 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.44.md b/doc/user/application_security/dast/checks/798.44.md new file mode 100644 index 00000000000..183cb49b2e7 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.44.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Finnhub Access Token + +## Description + +The response body contains content that matches the pattern of a Finnhub Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.44 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.45.md b/doc/user/application_security/dast/checks/798.45.md new file mode 100644 index 00000000000..a800063f15d --- /dev/null +++ b/doc/user/application_security/dast/checks/798.45.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Finicity Public Key + +## Description + +The response body contains content that matches the pattern of a Finicity Public Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.45 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.46.md b/doc/user/application_security/dast/checks/798.46.md new file mode 100644 index 00000000000..5bf658ff610 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.46.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Flutterwave Secret Key + +## Description + +The response body contains content that matches the pattern of a Flutterwave Secret Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.46 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.47.md b/doc/user/application_security/dast/checks/798.47.md new file mode 100644 index 00000000000..a6c7b974b7f --- /dev/null +++ b/doc/user/application_security/dast/checks/798.47.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Flutterwave Encryption Key + +## Description + +The response body contains content that matches the pattern of a Flutterwave Encryption Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.47 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.48.md b/doc/user/application_security/dast/checks/798.48.md new file mode 100644 index 00000000000..523232cb00c --- /dev/null +++ b/doc/user/application_security/dast/checks/798.48.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Frame.io API token + +## Description + +The response body contains content that matches the pattern of a Frame.io API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.48 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.49.md b/doc/user/application_security/dast/checks/798.49.md new file mode 100644 index 00000000000..ab7f39c2376 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.49.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Freshbooks Access Token + +## Description + +The response body contains content that matches the pattern of a Freshbooks Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.49 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.5.md b/doc/user/application_security/dast/checks/798.5.md new file mode 100644 index 00000000000..6d55dcf54df --- /dev/null +++ b/doc/user/application_security/dast/checks/798.5.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Airtable API Key + +## Description + +The response body contains content that matches the pattern of a Airtable API Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.5 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.50.md b/doc/user/application_security/dast/checks/798.50.md new file mode 100644 index 00000000000..f0d864db119 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.50.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token GoCardless API token + +## Description + +The response body contains content that matches the pattern of a GoCardless API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.50 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.51.md b/doc/user/application_security/dast/checks/798.51.md new file mode 100644 index 00000000000..f131d31ae65 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.51.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token GCP API key + +## Description + +The response body contains content that matches the pattern of a GCP API key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.51 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.52.md b/doc/user/application_security/dast/checks/798.52.md new file mode 100644 index 00000000000..0c4ea4a540b --- /dev/null +++ b/doc/user/application_security/dast/checks/798.52.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token GitHub Personal Access Token + +## Description + +The response body contains content that matches the pattern of a GitHub Personal Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.52 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.53.md b/doc/user/application_security/dast/checks/798.53.md new file mode 100644 index 00000000000..62a548be627 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.53.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token GitHub OAuth Access Token + +## Description + +The response body contains content that matches the pattern of a GitHub OAuth Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.53 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.54.md b/doc/user/application_security/dast/checks/798.54.md new file mode 100644 index 00000000000..d29677899a5 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.54.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token GitHub App Token + +## Description + +The response body contains content that matches the pattern of a GitHub App Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.54 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.55.md b/doc/user/application_security/dast/checks/798.55.md new file mode 100644 index 00000000000..4c3bd9147c0 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.55.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token GitHub Refresh Token + +## Description + +The response body contains content that matches the pattern of a GitHub Refresh Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.55 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.56.md b/doc/user/application_security/dast/checks/798.56.md new file mode 100644 index 00000000000..563ea1f91a8 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.56.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token GitLab Personal Access Token + +## Description + +The response body contains content that matches the pattern of a GitLab Personal Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.56 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.57.md b/doc/user/application_security/dast/checks/798.57.md new file mode 100644 index 00000000000..25b32953ebd --- /dev/null +++ b/doc/user/application_security/dast/checks/798.57.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Gitter Access Token + +## Description + +The response body contains content that matches the pattern of a Gitter Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.57 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.58.md b/doc/user/application_security/dast/checks/798.58.md new file mode 100644 index 00000000000..056bcb0820a --- /dev/null +++ b/doc/user/application_security/dast/checks/798.58.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token HashiCorp Terraform user/org API token + +## Description + +The response body contains content that matches the pattern of a HashiCorp Terraform user/org API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.58 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.59.md b/doc/user/application_security/dast/checks/798.59.md new file mode 100644 index 00000000000..b7e6b4fa32b --- /dev/null +++ b/doc/user/application_security/dast/checks/798.59.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Heroku API Key + +## Description + +The response body contains content that matches the pattern of a Heroku API Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.59 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.6.md b/doc/user/application_security/dast/checks/798.6.md new file mode 100644 index 00000000000..ce6ee95bede --- /dev/null +++ b/doc/user/application_security/dast/checks/798.6.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Algolia API Key + +## Description + +The response body contains content that matches the pattern of a Algolia API Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.6 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.60.md b/doc/user/application_security/dast/checks/798.60.md new file mode 100644 index 00000000000..f471411440b --- /dev/null +++ b/doc/user/application_security/dast/checks/798.60.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token HubSpot API Token + +## Description + +The response body contains content that matches the pattern of a HubSpot API Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.60 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.61.md b/doc/user/application_security/dast/checks/798.61.md new file mode 100644 index 00000000000..061bf8f7360 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.61.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Intercom API Token + +## Description + +The response body contains content that matches the pattern of a Intercom API Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.61 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.62.md b/doc/user/application_security/dast/checks/798.62.md new file mode 100644 index 00000000000..9c0f312b161 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.62.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Kraken Access Token + +## Description + +The response body contains content that matches the pattern of a Kraken Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.62 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.63.md b/doc/user/application_security/dast/checks/798.63.md new file mode 100644 index 00000000000..51668619025 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.63.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Kucoin Access Token + +## Description + +The response body contains content that matches the pattern of a Kucoin Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.63 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.64.md b/doc/user/application_security/dast/checks/798.64.md new file mode 100644 index 00000000000..12d20f96a42 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.64.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Kucoin Secret Key + +## Description + +The response body contains content that matches the pattern of a Kucoin Secret Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.64 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.65.md b/doc/user/application_security/dast/checks/798.65.md new file mode 100644 index 00000000000..eb1dac62037 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.65.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Launchdarkly Access Token + +## Description + +The response body contains content that matches the pattern of a Launchdarkly Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.65 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.66.md b/doc/user/application_security/dast/checks/798.66.md new file mode 100644 index 00000000000..8f20f9fa339 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.66.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Linear API Token + +## Description + +The response body contains content that matches the pattern of a Linear API Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.66 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.67.md b/doc/user/application_security/dast/checks/798.67.md new file mode 100644 index 00000000000..7554c077376 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.67.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Linear Client Secret + +## Description + +The response body contains content that matches the pattern of a Linear Client Secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.67 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.68.md b/doc/user/application_security/dast/checks/798.68.md new file mode 100644 index 00000000000..c633b949185 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.68.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token LinkedIn Client ID + +## Description + +The response body contains content that matches the pattern of a LinkedIn Client ID. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.68 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.69.md b/doc/user/application_security/dast/checks/798.69.md new file mode 100644 index 00000000000..b34c2f01be6 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.69.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token LinkedIn Client secret + +## Description + +The response body contains content that matches the pattern of a LinkedIn Client secret. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.69 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.7.md b/doc/user/application_security/dast/checks/798.7.md new file mode 100644 index 00000000000..43aba566471 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.7.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Alibaba AccessKey ID + +## Description + +The response body contains content that matches the pattern of a Alibaba AccessKey ID. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.7 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.70.md b/doc/user/application_security/dast/checks/798.70.md new file mode 100644 index 00000000000..b7c1816481b --- /dev/null +++ b/doc/user/application_security/dast/checks/798.70.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Lob API Key + +## Description + +The response body contains content that matches the pattern of a Lob API Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.70 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.71.md b/doc/user/application_security/dast/checks/798.71.md new file mode 100644 index 00000000000..f0bcc43940d --- /dev/null +++ b/doc/user/application_security/dast/checks/798.71.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Lob Publishable API Key + +## Description + +The response body contains content that matches the pattern of a Lob Publishable API Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.71 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.72.md b/doc/user/application_security/dast/checks/798.72.md new file mode 100644 index 00000000000..48b2cffbbda --- /dev/null +++ b/doc/user/application_security/dast/checks/798.72.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Mailchimp API key + +## Description + +The response body contains content that matches the pattern of a Mailchimp API key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.72 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.73.md b/doc/user/application_security/dast/checks/798.73.md new file mode 100644 index 00000000000..eae41a49782 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.73.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Mailgun public validation key + +## Description + +The response body contains content that matches the pattern of a Mailgun public validation key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.73 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.74.md b/doc/user/application_security/dast/checks/798.74.md new file mode 100644 index 00000000000..9a4b909bf4b --- /dev/null +++ b/doc/user/application_security/dast/checks/798.74.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Mailgun private API token + +## Description + +The response body contains content that matches the pattern of a Mailgun private API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.74 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.75.md b/doc/user/application_security/dast/checks/798.75.md new file mode 100644 index 00000000000..4c1cfd78003 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.75.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Mailgun webhook signing key + +## Description + +The response body contains content that matches the pattern of a Mailgun webhook signing key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.75 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.76.md b/doc/user/application_security/dast/checks/798.76.md new file mode 100644 index 00000000000..87e6364184f --- /dev/null +++ b/doc/user/application_security/dast/checks/798.76.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token MapBox API token + +## Description + +The response body contains content that matches the pattern of a MapBox API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.76 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.77.md b/doc/user/application_security/dast/checks/798.77.md new file mode 100644 index 00000000000..7b1becf4c19 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.77.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Mattermost Access Token + +## Description + +The response body contains content that matches the pattern of a Mattermost Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.77 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.78.md b/doc/user/application_security/dast/checks/798.78.md new file mode 100644 index 00000000000..8d366d44c9d --- /dev/null +++ b/doc/user/application_security/dast/checks/798.78.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token MessageBird API token + +## Description + +The response body contains content that matches the pattern of a MessageBird API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.78 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.79.md b/doc/user/application_security/dast/checks/798.79.md new file mode 100644 index 00000000000..9a580658a72 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.79.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token MessageBird client ID + +## Description + +The response body contains content that matches the pattern of a MessageBird client ID. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.79 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.8.md b/doc/user/application_security/dast/checks/798.8.md new file mode 100644 index 00000000000..e6dfe1aa1cc --- /dev/null +++ b/doc/user/application_security/dast/checks/798.8.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Alibaba Secret Key + +## Description + +The response body contains content that matches the pattern of a Alibaba Secret Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.8 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.80.md b/doc/user/application_security/dast/checks/798.80.md new file mode 100644 index 00000000000..c0a893264b0 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.80.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Netlify Access Token + +## Description + +The response body contains content that matches the pattern of a Netlify Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.80 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.81.md b/doc/user/application_security/dast/checks/798.81.md new file mode 100644 index 00000000000..abf40705e7f --- /dev/null +++ b/doc/user/application_security/dast/checks/798.81.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token New Relic user API Key + +## Description + +The response body contains content that matches the pattern of a New Relic user API Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.81 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.82.md b/doc/user/application_security/dast/checks/798.82.md new file mode 100644 index 00000000000..519555546b6 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.82.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token New Relic user API ID + +## Description + +The response body contains content that matches the pattern of a New Relic user API ID. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.82 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.83.md b/doc/user/application_security/dast/checks/798.83.md new file mode 100644 index 00000000000..85bdd534390 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.83.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token New Relic ingest browser API token + +## Description + +The response body contains content that matches the pattern of a New Relic ingest browser API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.83 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.84.md b/doc/user/application_security/dast/checks/798.84.md new file mode 100644 index 00000000000..74ebb4fcaf1 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.84.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token npm access token + +## Description + +The response body contains content that matches the pattern of a npm access token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.84 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.85.md b/doc/user/application_security/dast/checks/798.85.md new file mode 100644 index 00000000000..0726bdc7fd8 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.85.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Nytimes Access Token + +## Description + +The response body contains content that matches the pattern of a Nytimes Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.85 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.86.md b/doc/user/application_security/dast/checks/798.86.md new file mode 100644 index 00000000000..940a46b7658 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.86.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Okta Access Token + +## Description + +The response body contains content that matches the pattern of a Okta Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.86 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.87.md b/doc/user/application_security/dast/checks/798.87.md new file mode 100644 index 00000000000..8246bafc993 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.87.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Plaid Client ID + +## Description + +The response body contains content that matches the pattern of a Plaid Client ID. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.87 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.88.md b/doc/user/application_security/dast/checks/798.88.md new file mode 100644 index 00000000000..57b029857ba --- /dev/null +++ b/doc/user/application_security/dast/checks/798.88.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Plaid Secret key + +## Description + +The response body contains content that matches the pattern of a Plaid Secret key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.88 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.89.md b/doc/user/application_security/dast/checks/798.89.md new file mode 100644 index 00000000000..466044834dd --- /dev/null +++ b/doc/user/application_security/dast/checks/798.89.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Plaid API Token + +## Description + +The response body contains content that matches the pattern of a Plaid API Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.89 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.9.md b/doc/user/application_security/dast/checks/798.9.md new file mode 100644 index 00000000000..12c725cfd08 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.9.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Asana Client ID + +## Description + +The response body contains content that matches the pattern of a Asana Client ID. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.9 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.90.md b/doc/user/application_security/dast/checks/798.90.md new file mode 100644 index 00000000000..e0008af4918 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.90.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token PlanetScale password + +## Description + +The response body contains content that matches the pattern of a PlanetScale password. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.90 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.91.md b/doc/user/application_security/dast/checks/798.91.md new file mode 100644 index 00000000000..be54e99360f --- /dev/null +++ b/doc/user/application_security/dast/checks/798.91.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token PlanetScale API token + +## Description + +The response body contains content that matches the pattern of a PlanetScale API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.91 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.92.md b/doc/user/application_security/dast/checks/798.92.md new file mode 100644 index 00000000000..07ae24151f5 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.92.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token PlanetScale OAuth token + +## Description + +The response body contains content that matches the pattern of a PlanetScale OAuth token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.92 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.93.md b/doc/user/application_security/dast/checks/798.93.md new file mode 100644 index 00000000000..661f460bf27 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.93.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Postman API token + +## Description + +The response body contains content that matches the pattern of a Postman API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.93 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.94.md b/doc/user/application_security/dast/checks/798.94.md new file mode 100644 index 00000000000..4aeb15fee23 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.94.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Private Key + +## Description + +The response body contains content that matches the pattern of a Private Key. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.94 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.95.md b/doc/user/application_security/dast/checks/798.95.md new file mode 100644 index 00000000000..13374aa67e0 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.95.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Pulumi API token + +## Description + +The response body contains content that matches the pattern of a Pulumi API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.95 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.96.md b/doc/user/application_security/dast/checks/798.96.md new file mode 100644 index 00000000000..cb61bd38950 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.96.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token PyPI upload token + +## Description + +The response body contains content that matches the pattern of a PyPI upload token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.96 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.97.md b/doc/user/application_security/dast/checks/798.97.md new file mode 100644 index 00000000000..93f03a692d7 --- /dev/null +++ b/doc/user/application_security/dast/checks/798.97.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Rubygem API token + +## Description + +The response body contains content that matches the pattern of a Rubygem API token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.97 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.98.md b/doc/user/application_security/dast/checks/798.98.md new file mode 100644 index 00000000000..aab4cb9c5ed --- /dev/null +++ b/doc/user/application_security/dast/checks/798.98.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token RapidAPI Access Token + +## Description + +The response body contains content that matches the pattern of a RapidAPI Access Token. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.98 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/798.99.md b/doc/user/application_security/dast/checks/798.99.md new file mode 100644 index 00000000000..90c8aeda7ab --- /dev/null +++ b/doc/user/application_security/dast/checks/798.99.md @@ -0,0 +1,26 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Exposure of confidential secret or token Sendbird Access ID + +## Description + +The response body contains content that matches the pattern of a Sendbird Access ID. +Exposing this value could allow attackers to gain access to all resources granted by this token. + +## Remediation + +Review the response body content and remove any exposed values. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 798.99 | false | 798 | Passive | High | + +## Links + +- [CWE](https://cwe.mitre.org/data/definitions/798.html) diff --git a/doc/user/application_security/dast/checks/index.md b/doc/user/application_security/dast/checks/index.md index e2947d5b120..cdfebc07ef2 100644 --- a/doc/user/application_security/dast/checks/index.md +++ b/doc/user/application_security/dast/checks/index.md @@ -12,14 +12,20 @@ The [DAST browser-based crawler](../browser_based.md) provides a number of vulne |:---|:------|:---------|:-----| | [1004.1](1004.1.md) | Sensitive cookie without HttpOnly attribute | Low | Passive | | [16.1](16.1.md) | Missing Content-Type header | Low | Passive | +| [16.10](16.10.md) | Content-Security-Policy violations | Info | Passive | | [16.2](16.2.md) | Server header exposes version information | Low | Passive | | [16.3](16.3.md) | X-Powered-By header exposes version information | Low | Passive | | [16.4](16.4.md) | X-Backend-Server header exposes server information | Info | Passive | | [16.5](16.5.md) | AspNet header exposes version information | Low | Passive | | [16.6](16.6.md) | AspNetMvc header exposes version information | Low | Passive | | [16.7](16.7.md) | Strict-Transport-Security header missing or invalid | Low | Passive | +| [16.8](16.8.md) | Content-Security-Policy analysis | Info | Passive | +| [16.9](16.9.md) | Content-Security-Policy-Report-Only analysis | Info | Passive | | [200.1](200.1.md) | Exposure of sensitive information to an unauthorized actor (private IP address) | Low | Passive | | [209.1](209.1.md) | Generation of error message containing sensitive information | Low | Passive | +| [209.2](209.2.md) | Generation of database error message containing sensitive information | Low | Passive | +| [287.1](287.1.md) | Insecure authentication over HTTP (Basic Authentication) | Medium | Passive | +| [287.2](287.2.md) | Insecure authentication over HTTP (Digest Authentication) | Low | Passive | | [319.1](319.1.md) | Mixed Content | Info | Passive | | [352.1](352.1.md) | Absence of anti-CSRF tokens | Medium | Passive | | [359.1](359.1.md) | Exposure of Private Personal Information (PII) to an unauthorized actor (credit card) | Medium | Passive | @@ -31,5 +37,133 @@ The [DAST browser-based crawler](../browser_based.md) provides a number of vulne | [601.1](601.1.md) | URL redirection to untrusted site ('open redirect') | Low | Passive | | [614.1](614.1.md) | Sensitive cookie without Secure attribute | Low | Passive | | [693.1](693.1.md) | Missing X-Content-Type-Options: nosniff | Low | Passive | +| [798.1](798.1.md) | Exposure of confidential secret or token Adafruit API Key | High | Passive | +| [798.2](798.2.md) | Exposure of confidential secret or token Adobe Client ID (OAuth Web) | High | Passive | +| [798.3](798.3.md) | Exposure of confidential secret or token Adobe Client Secret | High | Passive | +| [798.4](798.4.md) | Exposure of confidential secret or token Age secret key | High | Passive | +| [798.5](798.5.md) | Exposure of confidential secret or token Airtable API Key | High | Passive | +| [798.6](798.6.md) | Exposure of confidential secret or token Algolia API Key | High | Passive | +| [798.7](798.7.md) | Exposure of confidential secret or token Alibaba AccessKey ID | High | Passive | +| [798.8](798.8.md) | Exposure of confidential secret or token Alibaba Secret Key | High | Passive | +| [798.9](798.9.md) | Exposure of confidential secret or token Asana Client ID | High | Passive | +| [798.10](798.10.md) | Exposure of confidential secret or token Asana Client Secret | High | Passive | +| [798.11](798.11.md) | Exposure of confidential secret or token Atlassian API token | High | Passive | +| [798.12](798.12.md) | Exposure of confidential secret or token AWS | High | Passive | +| [798.13](798.13.md) | Exposure of confidential secret or token Bitbucket Client ID | High | Passive | +| [798.14](798.14.md) | Exposure of confidential secret or token Bitbucket Client Secret | High | Passive | +| [798.15](798.15.md) | Exposure of confidential secret or token Bittrex Access Key | High | Passive | +| [798.16](798.16.md) | Exposure of confidential secret or token Bittrex Secret Key | High | Passive | +| [798.17](798.17.md) | Exposure of confidential secret or token Beamer API token | High | Passive | +| [798.18](798.18.md) | Exposure of confidential secret or token Codecov Access Token | High | Passive | +| [798.19](798.19.md) | Exposure of confidential secret or token Coinbase Access Token | High | Passive | +| [798.20](798.20.md) | Exposure of confidential secret or token Clojars API token | High | Passive | +| [798.21](798.21.md) | Exposure of confidential secret or token Confluent Access Token | High | Passive | +| [798.22](798.22.md) | Exposure of confidential secret or token Confluent Secret Key | High | Passive | +| [798.23](798.23.md) | Exposure of confidential secret or token Contentful delivery API token | High | Passive | +| [798.24](798.24.md) | Exposure of confidential secret or token Databricks API token | High | Passive | +| [798.25](798.25.md) | Exposure of confidential secret or token Datadog Access Token | High | Passive | +| [798.26](798.26.md) | Exposure of confidential secret or token Discord API key | High | Passive | +| [798.27](798.27.md) | Exposure of confidential secret or token Discord client ID | High | Passive | +| [798.28](798.28.md) | Exposure of confidential secret or token Discord client secret | High | Passive | +| [798.29](798.29.md) | Exposure of confidential secret or token Doppler API token | High | Passive | +| [798.30](798.30.md) | Exposure of confidential secret or token Dropbox API secret | High | Passive | +| [798.31](798.31.md) | Exposure of confidential secret or token Dropbox long lived API token | High | Passive | +| [798.32](798.32.md) | Exposure of confidential secret or token Dropbox short lived API token | High | Passive | +| [798.33](798.33.md) | Exposure of confidential secret or token Droneci Access Token | High | Passive | +| [798.34](798.34.md) | Exposure of confidential secret or token Duffel API token | High | Passive | +| [798.35](798.35.md) | Exposure of confidential secret or token Dynatrace API token | High | Passive | +| [798.36](798.36.md) | Exposure of confidential secret or token EasyPost API token | High | Passive | +| [798.37](798.37.md) | Exposure of confidential secret or token EasyPost test API token | High | Passive | +| [798.38](798.38.md) | Exposure of confidential secret or token Etsy Access Token | High | Passive | +| [798.39](798.39.md) | Exposure of confidential secret or token Facebook | High | Passive | +| [798.40](798.40.md) | Exposure of confidential secret or token Fastly API key | High | Passive | +| [798.41](798.41.md) | Exposure of confidential secret or token Finicity Client Secret | High | Passive | +| [798.42](798.42.md) | Exposure of confidential secret or token Finicity API token | High | Passive | +| [798.43](798.43.md) | Exposure of confidential secret or token Flickr Access Token | High | Passive | +| [798.44](798.44.md) | Exposure of confidential secret or token Finnhub Access Token | High | Passive | +| [798.45](798.45.md) | Exposure of confidential secret or token Finicity Public Key | High | Passive | +| [798.46](798.46.md) | Exposure of confidential secret or token Flutterwave Secret Key | High | Passive | +| [798.47](798.47.md) | Exposure of confidential secret or token Flutterwave Encryption Key | High | Passive | +| [798.48](798.48.md) | Exposure of confidential secret or token Frame.io API token | High | Passive | +| [798.49](798.49.md) | Exposure of confidential secret or token Freshbooks Access Token | High | Passive | +| [798.50](798.50.md) | Exposure of confidential secret or token GoCardless API token | High | Passive | +| [798.51](798.51.md) | Exposure of confidential secret or token GCP API key | High | Passive | +| [798.52](798.52.md) | Exposure of confidential secret or token GitHub Personal Access Token | High | Passive | +| [798.53](798.53.md) | Exposure of confidential secret or token GitHub OAuth Access Token | High | Passive | +| [798.54](798.54.md) | Exposure of confidential secret or token GitHub App Token | High | Passive | +| [798.55](798.55.md) | Exposure of confidential secret or token GitHub Refresh Token | High | Passive | +| [798.56](798.56.md) | Exposure of confidential secret or token GitLab Personal Access Token | High | Passive | +| [798.57](798.57.md) | Exposure of confidential secret or token Gitter Access Token | High | Passive | +| [798.58](798.58.md) | Exposure of confidential secret or token HashiCorp Terraform user/org API token | High | Passive | +| [798.59](798.59.md) | Exposure of confidential secret or token Heroku API Key | High | Passive | +| [798.60](798.60.md) | Exposure of confidential secret or token HubSpot API Token | High | Passive | +| [798.61](798.61.md) | Exposure of confidential secret or token Intercom API Token | High | Passive | +| [798.62](798.62.md) | Exposure of confidential secret or token Kraken Access Token | High | Passive | +| [798.63](798.63.md) | Exposure of confidential secret or token Kucoin Access Token | High | Passive | +| [798.64](798.64.md) | Exposure of confidential secret or token Kucoin Secret Key | High | Passive | +| [798.65](798.65.md) | Exposure of confidential secret or token Launchdarkly Access Token | High | Passive | +| [798.66](798.66.md) | Exposure of confidential secret or token Linear API Token | High | Passive | +| [798.67](798.67.md) | Exposure of confidential secret or token Linear Client Secret | High | Passive | +| [798.68](798.68.md) | Exposure of confidential secret or token LinkedIn Client ID | High | Passive | +| [798.69](798.69.md) | Exposure of confidential secret or token LinkedIn Client secret | High | Passive | +| [798.70](798.70.md) | Exposure of confidential secret or token Lob API Key | High | Passive | +| [798.71](798.71.md) | Exposure of confidential secret or token Lob Publishable API Key | High | Passive | +| [798.72](798.72.md) | Exposure of confidential secret or token Mailchimp API key | High | Passive | +| [798.73](798.73.md) | Exposure of confidential secret or token Mailgun public validation key | High | Passive | +| [798.74](798.74.md) | Exposure of confidential secret or token Mailgun private API token | High | Passive | +| [798.75](798.75.md) | Exposure of confidential secret or token Mailgun webhook signing key | High | Passive | +| [798.76](798.76.md) | Exposure of confidential secret or token MapBox API token | High | Passive | +| [798.77](798.77.md) | Exposure of confidential secret or token Mattermost Access Token | High | Passive | +| [798.78](798.78.md) | Exposure of confidential secret or token MessageBird API token | High | Passive | +| [798.79](798.79.md) | Exposure of confidential secret or token MessageBird client ID | High | Passive | +| [798.80](798.80.md) | Exposure of confidential secret or token Netlify Access Token | High | Passive | +| [798.81](798.81.md) | Exposure of confidential secret or token New Relic user API Key | High | Passive | +| [798.82](798.82.md) | Exposure of confidential secret or token New Relic user API ID | High | Passive | +| [798.83](798.83.md) | Exposure of confidential secret or token New Relic ingest browser API token | High | Passive | +| [798.84](798.84.md) | Exposure of confidential secret or token npm access token | High | Passive | +| [798.85](798.85.md) | Exposure of confidential secret or token Nytimes Access Token | High | Passive | +| [798.86](798.86.md) | Exposure of confidential secret or token Okta Access Token | High | Passive | +| [798.87](798.87.md) | Exposure of confidential secret or token Plaid Client ID | High | Passive | +| [798.88](798.88.md) | Exposure of confidential secret or token Plaid Secret key | High | Passive | +| [798.89](798.89.md) | Exposure of confidential secret or token Plaid API Token | High | Passive | +| [798.90](798.90.md) | Exposure of confidential secret or token PlanetScale password | High | Passive | +| [798.91](798.91.md) | Exposure of confidential secret or token PlanetScale API token | High | Passive | +| [798.92](798.92.md) | Exposure of confidential secret or token PlanetScale OAuth token | High | Passive | +| [798.93](798.93.md) | Exposure of confidential secret or token Postman API token | High | Passive | +| [798.94](798.94.md) | Exposure of confidential secret or token Private Key | High | Passive | +| [798.95](798.95.md) | Exposure of confidential secret or token Pulumi API token | High | Passive | +| [798.96](798.96.md) | Exposure of confidential secret or token PyPI upload token | High | Passive | +| [798.97](798.97.md) | Exposure of confidential secret or token Rubygem API token | High | Passive | +| [798.98](798.98.md) | Exposure of confidential secret or token RapidAPI Access Token | High | Passive | +| [798.99](798.99.md) | Exposure of confidential secret or token Sendbird Access ID | High | Passive | +| [798.100](798.100.md) | Exposure of confidential secret or token Sendbird Access Token | High | Passive | +| [798.101](798.101.md) | Exposure of confidential secret or token SendGrid API token | High | Passive | +| [798.102](798.102.md) | Exposure of confidential secret or token Sendinblue API token | High | Passive | +| [798.103](798.103.md) | Exposure of confidential secret or token Sentry Access Token | High | Passive | +| [798.104](798.104.md) | Exposure of confidential secret or token Shippo API token | High | Passive | +| [798.105](798.105.md) | Exposure of confidential secret or token Shopify access token | High | Passive | +| [798.106](798.106.md) | Exposure of confidential secret or token Shopify custom access token | High | Passive | +| [798.107](798.107.md) | Exposure of confidential secret or token Shopify private app access token | High | Passive | +| [798.108](798.108.md) | Exposure of confidential secret or token Shopify shared secret | High | Passive | +| [798.109](798.109.md) | Exposure of confidential secret or token Slack token | High | Passive | +| [798.110](798.110.md) | Exposure of confidential secret or token Slack Webhook | High | Passive | +| [798.111](798.111.md) | Exposure of confidential secret or token Stripe | High | Passive | +| [798.112](798.112.md) | Exposure of confidential secret or token Square Access Token | High | Passive | +| [798.113](798.113.md) | Exposure of confidential secret or token Squarespace Access Token | High | Passive | +| [798.114](798.114.md) | Exposure of confidential secret or token SumoLogic Access ID | High | Passive | +| [798.115](798.115.md) | Exposure of confidential secret or token SumoLogic Access Token | High | Passive | +| [798.116](798.116.md) | Exposure of confidential secret or token Travis CI Access Token | High | Passive | +| [798.117](798.117.md) | Exposure of confidential secret or token Twilio API Key | High | Passive | +| [798.118](798.118.md) | Exposure of confidential secret or token Twitch API token | High | Passive | +| [798.119](798.119.md) | Exposure of confidential secret or token Twitter API Key | High | Passive | +| [798.120](798.120.md) | Exposure of confidential secret or token Twitter API Secret | High | Passive | +| [798.121](798.121.md) | Exposure of confidential secret or token Twitter Access Token | High | Passive | +| [798.122](798.122.md) | Exposure of confidential secret or token Twitter Access Secret | High | Passive | +| [798.123](798.123.md) | Exposure of confidential secret or token Twitter Bearer Token | High | Passive | +| [798.124](798.124.md) | Exposure of confidential secret or token Typeform API token | High | Passive | +| [798.125](798.125.md) | Exposure of confidential secret or token Yandex API Key | High | Passive | +| [798.126](798.126.md) | Exposure of confidential secret or token Yandex AWS Access Token | High | Passive | +| [798.127](798.127.md) | Exposure of confidential secret or token Yandex Access Token | High | Passive | +| [798.128](798.128.md) | Exposure of confidential secret or token Zendesk Secret Key | High | Passive | | [829.1](829.1.md) | Inclusion of Functionality from Untrusted Control Sphere | Low | Passive | | [829.2](829.2.md) | Invalid Sub-Resource Integrity values detected | Medium | Passive | diff --git a/doc/user/application_security/dast/dast_troubleshooting.md b/doc/user/application_security/dast/dast_troubleshooting.md index 50570b89920..0c7a9806c72 100644 --- a/doc/user/application_security/dast/dast_troubleshooting.md +++ b/doc/user/application_security/dast/dast_troubleshooting.md @@ -102,3 +102,8 @@ To avoid this error, make sure you are using the latest stable version of Docker ## Lack of IPv6 support Due to the underlying [ZAProxy engine not supporting IPv6](https://github.com/zaproxy/zaproxy/issues/3705), DAST is unable to scan or crawl IPv6-based applications. + +## Additional insight into DAST scan activity + +For additional insight into what a DAST scan is doing at a given time, you may find it helpful to review +the web server access logs for a DAST target endpoint during or following a scan. diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 25b4b705025..f8aa2e3d1c6 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -90,7 +90,7 @@ deploy: services: - name: docker:dind alias: dind - image: docker:19.03.5 + image: docker:20.10.16 stage: build script: - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY @@ -99,9 +99,10 @@ deploy: - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA - docker push $CI_REGISTRY_IMAGE:latest -services: # use services to link your app container to the dast job - - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA - alias: yourapp +dast: + services: # use services to link your app container to the dast job + - name: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA + alias: yourapp variables: DAST_FULL_SCAN_ENABLED: "true" # do a full scan @@ -622,6 +623,11 @@ To enable Mutual TLS: These CI/CD variables are specific to DAST. They can be used to customize the behavior of DAST to your requirements. +WARNING: +All customization of GitLab security scanning tools should be tested in a merge request before +merging these changes to the default branch. Failure to do so can give unexpected results, +including a large number of false positives. + | CI/CD variable | Type | Description | |:-------------------------------------------------|:--------------|:------------------------------| | `DAST_ADVERTISE_SCAN` | boolean | Set to `true` to add a `Via` header to every request sent, advertising that the request was sent as part of a GitLab DAST scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334947) in GitLab 14.1. | diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index 9128576bf29..fdca02267e4 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -808,6 +808,9 @@ Optionally: - `DAST_API_PRE_SCRIPT`: Script to install runtimes or dependencies before the scan starts. +WARNING: +To execute scripts in Alpine Linux you must first use the command [`chmod`](https://www.gnu.org/software/coreutils/manual/html_node/chmod-invocation.html) to set the [execution permission](https://www.gnu.org/software/coreutils/manual/html_node/Setting-Permissions.html). For example, to set the execution permission of `script.py` for everyone, use the command: `chmod a+x script.py`. If needed, you can version your `script.py` with the execution permission already set. + ```yaml stages: - dast @@ -856,7 +859,9 @@ import requests import backoff # [1] Store log file in directory indicated by env var CI_PROJECT_DIR -working_directory = os.environ['CI_PROJECT_DIR'] +working_directory = os.environ.get( 'CI_PROJECT_DIR') +overrides_file_name = os.environ.get('DAST_API_OVERRIDES_FILE', 'dast-api-overrides.json') +overrides_file_path = os.path.join(working_directory, overrides_file_name) # [2] File name should match the pattern: gl-*.log log_file_path = os.path.join(working_directory, 'gl-user-overrides.log') @@ -870,7 +875,11 @@ logging.basicConfig(filename=log_file_path, level=logging.DEBUG) requests.exceptions.ConnectionError), max_time=30) def get_auth_response(): - return requests.get('https://authorization.service/api/get_api_token', auth=(os.environ['AUTH_USER'], os.environ['AUTH_PWD'])) + authorization_url = 'https://authorization.service/api/get_api_token' + return requests.get( + f'{authorization_url}', + auth=(os.environ.get('AUTH_USER'), os.environ.get('AUTH_PWD')) + ) # In our example, access token is retrieved from a given endpoint try: @@ -892,14 +901,14 @@ try: # requests.ReadTimeout : The server did not send any data in the allotted amount of time. # requests.TooManyRedirects : The request exceeds the configured number of maximum redirections # requests.exceptions.RequestException : All exceptions that related to Requests +except json.JSONDecodeError as json_decode_error: + # logs errors related decoding JSON response + logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}') + raise except requests.exceptions.RequestException as requests_error: # logs exceptions related to `Requests` logging.error(f'Error, failed while performing HTTP request. Error message: {requests_error}') raise -except requests.exceptions.JSONDecodeError as json_decode_error: - # logs errors related decoding JSON response - logging.error(f'Error, failed while decoding JSON response. Error message: {json_decode_error}') - raise except Exception as e: # logs any other error logging.error(f'Error, unknown error while retrieving access token. Error message: {e}') @@ -914,9 +923,6 @@ overrides_data = { } # log entry informing about the file override computation -# the location of the overrides json file is also CI_PROJECT_DIR -overrides_file_path = os.path.join( - working_directory, "dast-api-overrides.json") logging.info("Creating overrides file: %s" % overrides_file_path) # attempts to overwrite the file @@ -929,7 +935,7 @@ try: fd.write(json.dumps(overrides_data).encode('utf-8')) except Exception as e: # logs any other error - logging.error(f'Error, unkown error when overwritng file {overrides_file_path}. Error message: {e}') + logging.error(f'Error, unknown error when overwriting file {overrides_file_path}. Error message: {e}') raise # logs informing override has finished successfully diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 08e2dcd2e7e..d0a91ab664e 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -63,18 +63,19 @@ possible, we encourage you to use all of our security scanning tools: The following table summarizes which types of dependencies each scanning tool can detect: -| Feature | Dependency Scanning | Container Scanning | -| ----------------------------------------------------------- | ------------------- | ------------------ | -| Identify the manifest, lock file, or static file that introduced the dependency | **{check-circle}** | **{dotted-circle}** | -| Development dependencies | **{check-circle}** | **{dotted-circle}** | -| Dependencies in a lock file committed to your repository | **{check-circle}** | **{check-circle}** <sup>1</sup> | -| Binaries built by Go | **{dotted-circle}** | **{check-circle}** <sup>2</sup> | -| Dynamically-linked language-specific dependencies installed by the Operating System | **{dotted-circle}** | **{check-circle}** | -| Operating system dependencies | **{dotted-circle}** | **{check-circle}** | -| Language-specific dependencies installed on the operating system (not built by your project) | **{dotted-circle}** | **{check-circle}** | +| Feature | Dependency Scanning | Container Scanning | +| ----------------------------------------------------------- | ------------------- | ------------------- | +| Identify the manifest, lock file, or static file that introduced the dependency | **{check-circle}** | **{dotted-circle}** | +| Development dependencies | **{check-circle}** | **{dotted-circle}** | +| Dependencies in a lock file committed to your repository | **{check-circle}** | **{check-circle}** <sup>1</sup> | +| Binaries built by Go | **{dotted-circle}** | **{check-circle}** <sup>2</sup> <sup>3</sup> | +| Dynamically-linked language-specific dependencies installed by the Operating System | **{dotted-circle}** | **{check-circle}** <sup>3</sup> | +| Operating system dependencies | **{dotted-circle}** | **{check-circle}** | +| Language-specific dependencies installed on the operating system (not built by your project) | **{dotted-circle}** | **{check-circle}** | 1. Lock file must be present in the image to be detected. 1. Binary file must be present in the image to be detected. +1. Only when using Trivy ## Requirements @@ -310,7 +311,7 @@ table.supported-languages ul { <p> Although Gradle with Java 8 is supported, there are other issues such that Android project builds are not supported at this time. Please see the backlog issue <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/336866">Android support for Dependency - Scanning (gemnasium-maven)</a> for more details. + Scanning (gemnasium-maven)</a> for more details. Also, Gradle is not supported when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) is enabled. </p> </li> <li> @@ -396,11 +397,10 @@ To support the following package managers, the GitLab analyzers proceed in two s If your project <i>does not use</i> a <code>gradlew</code> file, then the analyzer automatically switches to one of the pre-installed Gradle versions, based on the version of Java specified by the <a href="#configuring-specific-analyzers-used-by-dependency-scanning"><code>DS_JAVA_VERSION</code></a> variable. + By default, the analyzer uses Java 17 and Gradle 7.3.3. </p> - <p>You can view the - <a href="https://docs.gradle.org/current/userguide/compatibility.html#java">Gradle Java compatibility matrix</a> to see which version - of Gradle is selected for each Java version. Note that we only support switching to one of these pre-installed Gradle versions - for Java versions 13 to 17. + <p> + For Java versions <code>8</code> and <code>11</code>, Gradle <code>6.7.1</code> is automatically selected, and for Java versions <code>13</code> to <code>17</code>, Gradle <code>7.3.3</code> is automatically selected. </p> </li> <li> @@ -587,6 +587,11 @@ gemnasium-dependency_scanning: Dependency scanning can be [configured](#customizing-the-dependency-scanning-settings) using environment variables. +WARNING: +All customization of GitLab security scanning tools should be tested in a merge request before +merging these changes to the default branch. Failure to do so can give unexpected results, +including a large number of false positives. + #### Configuring dependency scanning The following variables allow configuration of global dependency scanning settings. @@ -611,7 +616,7 @@ The following variables are used for configuring specific analyzers (used for a | `GEMNASIUM_DB_UPDATE_DISABLED` | `gemnasium` | `"false"` | Disable automatic updates for the `gemnasium-db` advisory database (For usage see: [examples](#hosting-a-copy-of-the-gemnasium_db-advisory-database))| | `GEMNASIUM_DB_REMOTE_URL` | `gemnasium` | `https://gitlab.com/gitlab-org/security-products/gemnasium-db.git` | Repository URL for fetching the Gemnasium database. | | `GEMNASIUM_DB_REF_NAME` | `gemnasium` | `master` | Branch name for remote repository database. `GEMNASIUM_DB_REMOTE_URL` is required. | -| `DS_REMEDIATE` | `gemnasium` | `"true"` | Enable automatic remediation of vulnerable dependencies. | +| `DS_REMEDIATE` | `gemnasium` | `"true"`, `"false"` in FIPS mode | Enable automatic remediation of vulnerable dependencies. Not supported in FIPS mode. | | `GEMNASIUM_LIBRARY_SCAN_ENABLED` | `gemnasium` | `"true"` | Enable detecting vulnerabilities in vendored JavaScript libraries. For now, `gemnasium` leverages [`Retire.js`](https://github.com/RetireJS/retire.js) to do this job. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350512) in GitLab 14.8. | | `DS_JAVA_VERSION` | `gemnasium-maven` | `17` | Version of Java. Available versions: `8`, `11`, `13`, `14`, `15`, `16`, `17`. Available versions in FIPS-enabled image: `8`, `11`, `17`. | | `MAVEN_CLI_OPTS` | `gemnasium-maven` | `"-DskipTests --batch-mode"` | List of command line arguments that are passed to `maven` by the analyzer. See an example for [using private repositories](../index.md#using-private-maven-repositories). | @@ -622,7 +627,7 @@ The following variables are used for configuring specific analyzers (used for a | `PIP_REQUIREMENTS_FILE` | `gemnasium-python` | | Pip requirements file to be scanned. | | `DS_PIP_VERSION` | `gemnasium-python` | | Force the install of a specific pip version (example: `"19.3"`), otherwise the pip installed in the Docker image is used. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12811) in GitLab 12.7) | | `DS_PIP_DEPENDENCY_PATH` | `gemnasium-python` | | Path to load Python pip dependencies from. ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12412) in GitLab 12.2) | -| `DS_INCLUDE_DEV_DEPENDENCIES` | `gemnasium` | `"true"` | When set to `"false"`, development dependencies and their vulnerabilities are not reported. Only NPM projects are supported. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227861) in GitLab 15.1. | +| `DS_INCLUDE_DEV_DEPENDENCIES` | `gemnasium` | `"true"` | When set to `"false"`, development dependencies and their vulnerabilities are not reported. Only NPM and Poetry projects are supported. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/227861) in GitLab 15.1. | #### Other variables @@ -688,6 +693,8 @@ To manually switch to FIPS-enabled images, set the variable `DS_IMAGE_SUFFIX` to To ensure compliance with FIPS, the FIPS-enabled image of `gemnasium-maven` uses the OpenJDK packages for RedHat UBI. As a result, it only supports Java 8, 11, and 17. +Dependency scanning for Gradle projects and auto-remediation for Yarn projects are not supported in FIPS mode. + ## Interacting with the vulnerabilities Once a vulnerability is found, you can interact with it. Read more on how to @@ -834,10 +841,16 @@ Here's an example dependency scanning report: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/350509) in GitLab 14.8 in [Beta](../../../policy/alpha-beta-support.md#beta-features). +NOTE: +CycloneDX SBOMs are a [Beta](../../../policy/alpha-beta-support.md#beta-features) feature, +and the reports are subject to change during the beta period. Do not build integrations +that rely on the format of these SBOMs staying consistent, as the format might change +before the feature is made generally available. + In addition to the [JSON report file](#reports-json-format), the [Gemnasium](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) Dependency Scanning tool outputs a [CycloneDX](https://cyclonedx.org/) Software Bill of Materials (SBOM) for each supported lock or build file it detects. These CycloneDX SBOMs are named -`cyclonedx-<package-type>-<package-manager>.json`, and are saved in the same directory +`gl-sbom-<package-type>-<package-manager>.cdx.json`, and are saved in the same directory as the detected lock or build files. For example, if your project has the following structure: @@ -860,16 +873,16 @@ Then the Gemnasium scanner generates the following CycloneDX SBOMs: . ├── ruby-project/ │ ├── Gemfile.lock -│ └── cyclonedx-gem-bundler.json +│ └── gl-sbom-gem-bundler.cdx.json ├── ruby-project-2/ │ ├── Gemfile.lock -│ └── cyclonedx-gem-bundler.json +│ └── gl-sbom-gem-bundler.cdx.json ├── php-project/ │ ├── composer.lock -│ └── cyclonedx-packagist-composer.json +│ └── gl-sbom-packagist-composer.cdx.json └── go-project/ ├── go.sum - └── cyclonedx-go-go.json + └── gl-sbom-go-go.cdx.json ``` The CycloneDX SBOMs can be downloaded [the same way as other job artifacts](../../../ci/pipelines/job_artifacts.md#download-job-artifacts). @@ -894,10 +907,10 @@ merge cyclonedx sboms: - wget https://github.com/CycloneDX/cyclonedx-cli/releases/download/v0.22.0/cyclonedx-linux-musl-x64 -O /usr/local/bin/cyclonedx-cli - chmod 755 /usr/local/bin/cyclonedx-cli - apk --update add --no-cache icu-dev libstdc++ - - find * -name "cyclonedx-*.json" -exec cyclonedx-cli merge --input-files {} --output-file cyclonedx-all.json + + - find * -name "gl-sbom-*.cdx.json" -exec cyclonedx-cli merge --input-files {} --output-file gl-sbom-all.cdx.json + artifacts: paths: - - cyclonedx-all.json + - gl-sbom-all.cdx.json ``` GitLab uses [CycloneDX Properties](https://cyclonedx.org/use-cases/#properties--name-value-store) @@ -905,12 +918,6 @@ to store implementation-specific details in the metadata of each CycloneDX SBOM, such as the location of build and lock files. If multiple CycloneDX SBOMs are merged together, this information is removed from the resulting merged file. -NOTE: -CycloneDX SBOMs are a [Beta](../../../policy/alpha-beta-support.md#beta-features) feature, -and the reports are subject to change during the beta period. Do not build integrations -that rely on the format of these SBOMs staying consistent, as the format might change -before the feature is made generally available. - ## Versioning and release process Please check the [Release Process documentation](https://gitlab.com/gitlab-org/security-products/release/blob/master/docs/release_process.md). diff --git a/doc/user/application_security/generate_test_vulnerabilities/index.md b/doc/user/application_security/generate_test_vulnerabilities/index.md new file mode 100644 index 00000000000..aafbebb91cd --- /dev/null +++ b/doc/user/application_security/generate_test_vulnerabilities/index.md @@ -0,0 +1,28 @@ +--- +type: reference, howto +stage: Secure +group: Threat Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Generate test vulnerabilities + +You can generate test vulnerabilities when you work on the [Vulnerability Report](../vulnerability_report/index.md). + +1. Go to `/-/profile/personal_access_tokens` and generate a personal access token with `api` permissions. +1. Go to your project page and find the project ID. You can find the project ID below the project title. +1. Open a terminal and go to the `gitlab/qa` directory. +1. Run the following command: + +```shell +GITLAB_QA_ACCESS_TOKEN=<your_personal_access_token> GITLAB_URL="http://localhost:3000" bundle exec rake vulnerabilities:setup\[<your_project_id>,<vulnerability_count>\] --trace +``` + +Make sure you do the following: + +- Replace `<your_personal_access_token>` with the token you generated in step one. +- Double check the `GITLAB_URL`. It should point to the running local instance. +- Replace `<your_project_id>` with the ID you obtained in step two. +- Replace `<vulnerability_count>` with the number of vulnerabilities you'd like to generate. + +The script creates the specified amount of vulnerabilities in the project. diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index d449fbb9a6c..e3a419ea771 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -114,6 +114,11 @@ While you cannot directly customize Auto DevOps, you can [include the Auto DevOp To enable all GitLab security scanning tools, with the option of customizing settings, add the GitLab CI/CD templates to your `.gitlab-ci.yml` file. +WARNING: +All customization of GitLab security scanning tools should be tested in a merge request before +merging these changes to the default branch. Failure to do so can give unexpected results, +including a large number of false positives. + To enable Static Application Security Testing, Dependency Scanning, License Scanning, and Secret Detection, add: @@ -217,9 +222,9 @@ From the merge request security widget, select **Expand** to unfold the widget, ## View security scan information in the pipeline Security tab -A pipeline's security tab lists all findings in the current branch. It includes new findings introduced by this branch and existing vulnerabilities that were already present when the branch was created. These results likely do not match the findings displayed in the Merge Request security widget as those do not include the existing vulnerabilities (with the exception of showing any existing vulnerabilities that are no longer detected in the feature branch). - -For more details, see [security tab](security_dashboard/index.md#view-vulnerabilities-in-a-pipeline). +A pipeline's security tab lists all findings in the current branch. It includes new findings introduced by this branch +and existing vulnerabilities already present when you created the branch. These results likely do not match the findings +displayed in the Merge Request security widget, as those do not include the existing vulnerabilities. Refer to [View vulnerabilities in a pipeline](vulnerability_report/pipeline.md) for more information. ## View security scan information in the Security Dashboard @@ -247,15 +252,6 @@ security issues: - A software license compliance violation. For more details, read [Enabling license approvals within a project](../compliance/license_compliance/index.md#enabling-license-approvals-within-a-project). -### Migration of existing Vulnerability-Check rules - -If your projects have rules that have a security orchestration project, a new MR with -the existing rule's content is created automatically against the default branch belonging -to the security orchestration project. To maintain the same security approval rules you -had before GitLab 15.0, we recommend merging this new MR. - -If your projects have rules without a security orchestration project, a new security orchestration project is created automatically with the content of the existing rule. No additional action is required. - ## Using private Maven repositories If you have a private Apache Maven repository that requires login credentials, @@ -393,48 +389,31 @@ Self managed installations can also run the security scanners on a GitLab Runner > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321918) in GitLab 13.11. > - Schema validation message [added](https://gitlab.com/gitlab-org/gitlab/-/issues/321730) in GitLab 14.0. -You can enforce validation of the security report artifacts before ingesting the vulnerabilities. +GitLab 15.0 enforces validation of the security report artifacts before ingesting the vulnerabilities. This prevents ingestion of broken vulnerability data into the database. GitLab validates the -artifacts based on the [report schemas](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/tree/master/dist). -When artifact validation is enabled, the pipeline's **Security** tab lists -any report artifacts that failed validation. +artifacts against the [report schemas](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/tree/master/dist), +according to the schema version declared in the report. -Validation depends on the schema: +The pipeline's **Security** tab lists any report artifacts that failed validation, and the +validation error message. -- If your security report does not specify which schema version it uses, GitLab attempts to verify it against the earliest supported schema version for that report type. Validation fails but it's attempted anyway because it may identify other problems present in the report. -- If your security report uses a version that is not supported, GitLab attempts to validate it against the earliest supported schema version for that report type. Validation fails but will identify the differences between the schema version used and the earliest supported version. -- If your security report uses a deprecated version, GitLab attempts validation against that version and adds a warning to the validation result. +Validation depends on the schema version declared in the security report artifact: -You can always find supported and deprecated schema versions in the [source code](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/parsers/security/validators/schema_validator.rb#L9). +- If your security report specifies a supported schema version, GitLab uses this version to validate. +- If your security report uses a deprecated version, GitLab attempts validation against that version and adds a deprecation warning to the validation result. +- If your security report uses a version that is not supported, GitLab attempts to validate it against the latest schema version available in GitLab. +- If your security report does not specify a schema version, GitLab attempts to validate it against the lastest schema version available in GitLab. Since the `version` property is required, validation always fails in this case, but other validation errors may also be present. -### Enable security report validation +You can always find supported and deprecated schema versions in the [source code](https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/parsers/security/validators/schema_validator.rb). -> [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/354928) in GitLab 14.9, and planned for removal in GitLab 15.0. +<!--- start_remove The following content will be removed on remove_date: '2022-08-22' --> -To enable report artifacts validation, set the `VALIDATE_SCHEMA` environment variable to `"true"` -for the desired jobs in the `.gitlab-ci.yml` file. +### Enable security report validation (removed) -For example, to enable validation for only the `sast` job: + This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/354928) in GitLab 14.9 + and [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/85400) in GitLab 15.0. -```yaml -include: - - template: Security/Dependency-Scanning.gitlab-ci.yml - - template: Security/License-Scanning.gitlab-ci.yml - - template: Security/SAST.gitlab-ci.yml - - template: Security/Secret-Detection.gitlab-ci.yml -stages: - - security-scan -dependency_scanning: - stage: security-scan -license_scanning: - stage: security-scan -sast: - stage: security-scan - variables: - VALIDATE_SCHEMA: "true" -.secret-analyzer: - stage: security-scan -``` + <!--- end_remove --> ## Interact with findings and vulnerabilities @@ -488,17 +467,16 @@ GitLab provides two methods of accomplishing this, each with advantages and disa - [Compliance framework pipelines](../project/settings/#compliance-pipeline-configuration) are recommended when: - - Scan execution enforcement is required for SAST or Secret Detection scans that use custom rulesets. - - Scan execution enforcement is required for SAST IaC, Dependency Scanning, + - Scan execution enforcement is required for any scanner that uses a GitLab template, such as SAST IaC, DAST, Dependency Scanning, License Compliance, API Fuzzing, or Coverage-guided Fuzzing. - Scan execution enforcement is required for scanners external to GitLab. - - Enforced execution is required for custom jobs other than security scans. + - Scan execution enforcement is required for custom jobs other than security scans. - [Scan execution policies](policies/scan-execution-policies.md) are recommended when: - - Scan execution enforcement is required for DAST. - - Scan execution enforcement is required for Container Scanning with project-specific variable + - Scan execution enforcement is required for DAST which uses a DAST site or scan profile. + - Scan execution enforcement is required for SAST, Secret Detection, or Container Scanning with project-specific variable customizations. To accomplish this, users must create a separate security policy per project. - Scans are required to run on a regular, scheduled cadence. @@ -524,6 +502,8 @@ Feedback is welcome on our vision for [unifying the user experience for these tw ## Troubleshooting +<!-- NOTE: The below subsection(`### Secure job failing with exit code 1`) documentation URL is referred in the [/gitlab-org/security-products/analyzers/command](https://gitlab.com/gitlab-org/security-products/analyzers/command/-/blob/main/command.go#L19) repository. If this section/subsection changes, please ensure to update the corresponding URL in the mentioned repository. +--> ### Secure job failing with exit code 1 If a Secure job is failing and it's unclear why, add `SECURE_LOG_LEVEL: "debug"` as a global CI/CD variable for diff --git a/doc/user/application_security/policies/img/policies_list_v15_0.png b/doc/user/application_security/policies/img/policies_list_v15_0.png Binary files differdeleted file mode 100644 index 4089c311fe4..00000000000 --- a/doc/user/application_security/policies/img/policies_list_v15_0.png +++ /dev/null diff --git a/doc/user/application_security/policies/img/policies_list_v15_1.png b/doc/user/application_security/policies/img/policies_list_v15_1.png Binary files differnew file mode 100644 index 00000000000..23c79a867ec --- /dev/null +++ b/doc/user/application_security/policies/img/policies_list_v15_1.png diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index 27a6f867ae2..81a9cef885d 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -81,7 +81,7 @@ status), and create and edit deployed policies: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > Policies**. -![Policies List Page](img/policies_list_v15_0.png) +![Policies List Page](img/policies_list_v15_1.png) ## Policy editor diff --git a/doc/user/application_security/policies/scan-execution-policies.md b/doc/user/application_security/policies/scan-execution-policies.md index 5beb6912877..eb1f9a7c7b8 100644 --- a/doc/user/application_security/policies/scan-execution-policies.md +++ b/doc/user/application_security/policies/scan-execution-policies.md @@ -6,10 +6,13 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Scan execution policies **(ULTIMATE)** -Project owners can use scan execution policies to require that security scans run on a specified -schedule or with the project pipeline. Required scans are injected into the CI pipeline as new jobs +> Group-level security policies were [introduced](https://gitlab.com/groups/gitlab-org/-/epics/4425) in GitLab 15.2 [with a flag](../../../administration/feature_flags.md) named `group_level_security_policies`. Enabled by default. + +Group, sub-group, or project owners can use scan execution policies to require that security scans run on a specified +schedule or with the project (or multiple projects if the policy is defined at a group or sub-group level) pipeline. Required scans are injected into the CI pipeline as new jobs with a long, random job name. In the unlikely event of a job name collision, the security policy job overwrites -any pre-existing job in the pipeline. +any pre-existing job in the pipeline. If a policy is created at the group-level, it will apply to every child +project or sub-group. A group-level policy cannot be edited from a child project or sub-group. This feature has some overlap with [compliance framework pipelines](../../project/settings/#compliance-pipeline-configuration), as we have not [unified the user experience for these two features](https://gitlab.com/groups/gitlab-org/-/epics/7312). @@ -25,7 +28,7 @@ an error appears that states `chosen stage does not exist`. ## Scan execution policy editor NOTE: -Only project Owners have the [permissions](../../permissions.md#project-members-permissions) +Only group, sub-group, or project Owners have the [permissions](../../permissions.md#project-members-permissions) to select Security Policy Project. Once your policy is complete, save it by selecting **Create via merge request** @@ -62,7 +65,7 @@ the following sections and tables provide an alternative. | Field | Type | Possible values | Description | |-------|------|-----------------|-------------| -| `name` | `string` | | Name of the policy. | +| `name` | `string` | | Name of the policy. Maximum of 255 characters.| | `description` (optional) | `string` | | Description of the policy. | | `enabled` | `boolean` | `true`, `false` | Flag to enable (`true`) or disable (`false`) the policy. | | `rules` | `array` of rules | | List of rules that the policy applies. | @@ -85,9 +88,8 @@ This rule enforces the defined actions and schedules a scan on the provided date |------------|------|-----------------|-------------| | `type` | `string` | `schedule` | The rule's type. | | `branches` | `array` of `string` | `*` or the branch's name | The branch the given policy applies to (supports wildcard). | -| `cadence` | `string` | CRON expression (for example, `0 0 * * *`) | A whitespace-separated string containing five fields that represents the scheduled time. | -| `agents` | `object` | | The name of the [GitLab agents](../../clusters/agent/index.md) where [cluster image scanning](../../clusters/agent/vulnerabilities.md) will run. The key of the object is the name of the Kubernetes cluster configured for your project in GitLab. In the optionally provided value of the object, you can precisely select Kubernetes resources that are scanned. <!--- start_remove The following content will be removed on remove_date: '2022-08-22' --> | -| `clusters` (removed) | `object` | | This field was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/356465) in 15.0. Use the `agents` field instead. The cluster where the given policy enforces running selected scans (only for `container_scanning`/`cluster_image_scanning` scans). The key of the object is the name of the Kubernetes cluster configured for your project in GitLab. In the optionally provided value of the object, you can precisely select Kubernetes resources that are scanned. <!--- end_remove --> | +| `cadence` | `string` | CRON expression (for example, `0 0 * * *`) | A whitespace-separated string containing five fields that represents the scheduled time. <!--- start_remove The following content will be removed on remove_date: '2022-08-22' --> | +| `clusters` (removed) | `object` | | This field was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/356465) in 15.0. The cluster where the given policy enforces running selected scans (only for `container_scanning`/`cluster_image_scanning` scans). The key of the object is the name of the Kubernetes cluster configured for your project in GitLab. In the optionally provided value of the object, you can precisely select Kubernetes resources that are scanned. <!--- end_remove --> | GitLab supports the following types of CRON syntax for the `cadence` field: @@ -96,20 +98,11 @@ GitLab supports the following types of CRON syntax for the `cadence` field: It is possible that other elements of the CRON syntax will work in the cadence field, however, GitLab does not officially test or support them. -### `agent` schema - -Use this schema to define `agents` objects in the [`schedule` rule type](#schedule-rule-type). - -| Field | Type | Possible values | Description | -|--------------|---------------------|--------------------------|-------------| -| `namespaces` | `array` of `string` | | The namespace that is scanned. If empty, all namespaces will be scanned. | - <!--- start_remove The following content will be removed on remove_date: '2022-08-22' --> ### `cluster` schema (removed) This schema was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/356465) in 15.0. -Use the [`agent` schema](#agent-schema) instead. Use this schema to define `clusters` objects in the [`schedule` rule type](#schedule-rule-type). @@ -200,24 +193,6 @@ scan_execution_policy: variables: SAST_EXCLUDED_ANALYZERS: brakeman - scan: container_scanning -- name: Enforce Cluster Image Scanning on production-cluster every 24h - description: This policy enforces Cluster Image Scanning scan to run every 24 hours - enabled: true - rules: - - type: schedule - cadence: "15 3 * * *" - clusters: - production-cluster: - containers: - - database - resources: - - production-application - namespaces: - - production-namespace - kinds: - - deployment - actions: - - scan: cluster_image_scanning ``` In this example: diff --git a/doc/user/application_security/policies/scan-result-policies.md b/doc/user/application_security/policies/scan-result-policies.md index 3da884aca6a..3eee4957e2f 100644 --- a/doc/user/application_security/policies/scan-result-policies.md +++ b/doc/user/application_security/policies/scan-result-policies.md @@ -56,7 +56,7 @@ the following sections and tables provide an alternative. | Field | Type | Possible values | Description | |-------|------|-----------------|-------------| -| `name` | `string` | | Name of the policy. | +| `name` | `string` | | Name of the policy. Maximum of 255 characters.| | `description` (optional) | `string` | | Description of the policy. | | `enabled` | `boolean` | `true`, `false` | Flag to enable (`true`) or disable (`false`) the policy. | | `rules` | `array` of rules | | List of rules that the policy applies. | @@ -69,7 +69,7 @@ This rule enforces the defined actions based on the information provided. | Field | Type | Possible values | Description | |------------|------|-----------------|-------------| | `type` | `string` | `scan_finding` | The rule's type. | -| `branches` | `array` of `string` | `[]` or the branch's name | Protected branches for this rule to consider. | +| `branches` | `array` of `string` | `[]` or the branch's name | Applicable only to protected target branches. An empty array, `[]`, applies the rule to all protected target branches. | | `scanners` | `array` of `string` | `sast`, `secret_detection`, `dependency_scanning`, `container_scanning`, `dast`, `coverage_fuzzing`, `api_fuzzing` | The security scanners for this rule to consider. | | `vulnerabilities_allowed` | `integer` | Greater than or equal to zero | Number of vulnerabilities allowed before this rule is considered. | | `severity_levels` | `array` of `string` | `info`, `unknown`, `low`, `medium`, `high`, `critical`| The severity levels for this rule to consider. | @@ -84,10 +84,10 @@ the defined policy. |-------|------|-----------------|-------------| | `type` | `string` | `require_approval` | The action's type. | | `approvals_required` | `integer` | Greater than or equal to zero | The number of MR approvals required. | -| `user_approvers` | `array` of `string` | Username of one of more users | The users to consider as approvers. | -| `user_approvers_ids` | `array` of `integer` | ID of one of more users | The IDs of users to consider as approvers. | -| `group_approvers` | `array` of `string` | Path of one of more groups | The groups to consider as approvers. | -| `group_approvers_ids` | `array` of `integer` | ID of one of more groups | The IDs of groups to consider as approvers. | +| `user_approvers` | `array` of `string` | Username of one of more users | The users to consider as approvers. Users must have access to the project to be eligible to approve. | +| `user_approvers_ids` | `array` of `integer` | ID of one of more users | The IDs of users to consider as approvers. Users must have access to the project to be eligible to approve. | +| `group_approvers` | `array` of `string` | Path of one of more groups | The groups to consider as approvers. Users with [direct membership in the group](../../project/merge_requests/approvals/rules.md#group-approvers) are eligible to approve. | +| `group_approvers_ids` | `array` of `integer` | ID of one of more groups | The IDs of groups to consider as approvers. Users with [direct membership in the group](../../project/merge_requests/approvals/rules.md#group-approvers) are eligible to approve. | Requirements and limitations: diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index d4dd8059c6a..92dc795afe5 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -74,41 +74,42 @@ GitLab SAST supports a variety of languages, package managers, and frameworks. O You can also [view our language roadmap](https://about.gitlab.com/direction/secure/static-analysis/sast/#language-support) and [request other language support by opening an issue](https://gitlab.com/groups/gitlab-org/-/epics/297). -| Language (package managers) / framework | Scan tool | Introduced in GitLab Version | -|---------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| -| .NET Core | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | -| .NET Framework | [Security Code Scan](https://security-code-scan.github.io) | 13.0 | -| Apex (Salesforce) | [PMD](https://pmd.github.io/pmd/index.html) | 12.1 | -| C | [Semgrep](https://semgrep.dev) | 14.2 | -| C/C++ | [Flawfinder](https://github.com/david-a-wheeler/flawfinder) | 10.7 | -| Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.1 | -| Go | [Gosec](https://github.com/securego/gosec) | 10.7 | -| Go | [Semgrep](https://semgrep.dev) | 14.4 | -| Groovy ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Ant, Maven, SBT) | -| Helm Charts | [Kubesec](https://github.com/controlplaneio/kubesec) | 13.1 | -| Java (any build system) | [Semgrep](https://semgrep.dev) | 14.10 | -| Java ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (Ant, SBT) | -| Java (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| JavaScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8 | -| JavaScript | [Semgrep](https://semgrep.dev) | 13.10 | -| Kotlin (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| Kotlin (General) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 13.11 | -| Kubernetes manifests | [Kubesec](https://github.com/controlplaneio/kubesec) | 12.6 | -| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | 11.1 | -| Objective-C (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | 10.8 | -| Python ([pip](https://pip.pypa.io/en/stable/)) | [bandit](https://github.com/PyCQA/bandit) | 10.3 | -| Python | [Semgrep](https://semgrep.dev) | 13.9 | -| React | [ESLint react plugin](https://github.com/yannickcr/eslint-plugin-react) | 12.5 | -| React | [Semgrep](https://semgrep.dev) | 13.10 | -| Ruby | [brakeman](https://brakemanscanner.org) | 13.9 | -| Ruby on Rails | [brakeman](https://brakemanscanner.org) | 10.3 | -| Scala ([Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/)) | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.0 (SBT) & 11.9 (Ant, Gradle, Maven) | -| Swift (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | -| TypeScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | -| TypeScript | [Semgrep](https://semgrep.dev) | 13.10 | - -Note that the Java analyzers can also be used for variants like the +| Language (package managers) / framework | Scan tool | Introduced in GitLab Version | +|------------------------------------------------|-----------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------| +| .NET Core | [Security Code Scan](https://security-code-scan.github.io) | 11.0 | +| .NET Framework<sup>1</sup> | [Security Code Scan](https://security-code-scan.github.io) | 13.0 | +| Apex (Salesforce) | [PMD](https://pmd.github.io/pmd/index.html) | 12.1 | +| C | [Semgrep](https://semgrep.dev) | 14.2 | +| C/C++ | [Flawfinder](https://github.com/david-a-wheeler/flawfinder) | 10.7 | +| Elixir (Phoenix) | [Sobelow](https://github.com/nccgroup/sobelow) | 11.1 | +| Go | [Gosec](https://github.com/securego/gosec) | 10.7 | +| Go | [Semgrep](https://semgrep.dev) | 14.4 | +| Groovy<sup>2</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.3 (Gradle) & 11.9 (Ant, Maven, SBT) | +| Helm Charts | [Kubesec](https://github.com/controlplaneio/kubesec) | 13.1 | +| Java (any build system) | [Semgrep](https://semgrep.dev) | 14.10 | +| Java<sup>2</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 10.6 (Maven), 10.8 (Gradle) & 11.9 (Ant, SBT) | +| Java (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | +| JavaScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.8 | +| JavaScript | [Semgrep](https://semgrep.dev) | 13.10 | +| Kotlin (Android) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | +| Kotlin (General)<sup>2</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 13.11 | +| Kubernetes manifests | [Kubesec](https://github.com/controlplaneio/kubesec) | 12.6 | +| Node.js | [NodeJsScan](https://github.com/ajinabraham/NodeJsScan) | 11.1 | +| Objective-C (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | +| PHP | [phpcs-security-audit](https://github.com/FloeDesignTechnologies/phpcs-security-audit) | 10.8 | +| Python ([pip](https://pip.pypa.io/en/stable/)) | [bandit](https://github.com/PyCQA/bandit) | 10.3 | +| Python | [Semgrep](https://semgrep.dev) | 13.9 | +| React | [ESLint react plugin](https://github.com/yannickcr/eslint-plugin-react) | 12.5 | +| React | [Semgrep](https://semgrep.dev) | 13.10 | +| Ruby | [brakeman](https://brakemanscanner.org) | 13.9 | +| Ruby on Rails | [brakeman](https://brakemanscanner.org) | 10.3 | +| Scala<sup>2</sup> | [SpotBugs](https://spotbugs.github.io/) with the [find-sec-bugs](https://find-sec-bugs.github.io/) plugin | 11.0 (SBT) & 11.9 (Ant, Gradle, Maven) | +| Swift (iOS) | [MobSF (beta)](https://github.com/MobSF/Mobile-Security-Framework-MobSF) | 13.5 | +| TypeScript | [ESLint security plugin](https://github.com/nodesecurity/eslint-plugin-security) | 11.9, [merged](https://gitlab.com/gitlab-org/gitlab/-/issues/36059) with ESLint in 13.2 | +| TypeScript | [Semgrep](https://semgrep.dev) | 13.10 | + +1. .NET 4 support is limited. The analyzer runs in a Linux container and does not have access to Windows-specific libraries or features. We currently plan to [migrate C# coverage to Semgrep-based scanning](https://gitlab.com/gitlab-org/gitlab/-/issues/347258) to make it easier to scan C# projects. +1. The SpotBugs-based analyzer supports [Ant](https://ant.apache.org/), [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/), and [SBT](https://www.scala-sbt.org/). It can also be used with variants like the [Gradle wrapper](https://docs.gradle.org/current/userguide/gradle_wrapper.html), [Grails](https://grails.org/), and the [Maven wrapper](https://github.com/takari/maven-wrapper). @@ -146,8 +147,8 @@ The default scanner images are build off a base Alpine image for size and mainta > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6479) in GitLab 14.10. -GitLab offers [Red Hat UBI](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) -versions of the images that are FIPS-enabled. To use the FIPS-enabled images, you can either: +GitLab offers an image version, based on the [Red Hat UBI](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) base image, +that uses a FIPS 140-validated cryptographic module. To use the FIPS-enabled image, you can either: - Set the `SAST_IMAGE_SUFFIX` to `-fips`. - Add the `-fips` extension to the default image name. @@ -162,6 +163,10 @@ include: - template: Security/SAST.gitlab-ci.yml ``` +A FIPS-compliant image is only available for the Semgrep-based analyzer. + +To use SAST in a FIPS-compliant manner, you must [exclude other analyzers from running](analyzers.md#customize-analyzers). + ### Making SAST analyzers available to all GitLab tiers All open source (OSS) analyzers have been moved to the GitLab Free tier as of GitLab 13.3. @@ -836,6 +841,11 @@ spotbugs-sast: SAST can be configured using the [`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. +WARNING: +All customization of GitLab security scanning tools should be tested in a merge request before +merging these changes to the default branch. Failure to do so can give unexpected results, +including a large number of false positives. + The following example includes the SAST template to override the `SAST_GOSEC_LEVEL` variable to `2`. The template is [evaluated before](../../../ci/yaml/index.md#include) the pipeline configuration, so the last mention of the variable takes precedence. diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index 9805fb3b67c..02d50b0a857 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -157,6 +157,11 @@ The Secret Detection scan settings can be changed through [CI/CD variables](#ava by using the [`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. +WARNING: +All customization of GitLab security scanning tools should be tested in a merge request before +merging these changes to the default branch. Failure to do so can give unexpected results, +including a large number of false positives. + To override a job definition, (for example, change properties like `variables` or `dependencies`), declare a job with the same name as the secret detection job to override. Place this new job after the template inclusion and specify any additional keys under it. diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index 3cb4bd4a02d..f3c834e06c7 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -42,57 +42,6 @@ To reduce false negatives in [dependency scans](../../../user/application_securi - Python projects can have lock files, but GitLab Secure tools don't support them. - Configure your project for [Continuous Delivery](../../../ci/introduction/index.md). -## View vulnerabilities in a pipeline - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13496) in GitLab 12.3. - -To view vulnerabilities in a pipeline: - -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **CI/CD > Pipelines**. -1. From the list, select the pipeline you want to check for vulnerabilities. -1. Select the **Security** tab. - -**Scan details** shows vulnerabilities introduced by the merge request, in addition to existing vulnerabilities -from the latest successful pipeline in your project's default branch. - -A pipeline consists of multiple jobs, such as SAST and DAST scans. If a job fails to finish, -the security dashboard doesn't show SAST scanner output. For example, if the SAST -job finishes but the DAST job fails, the security dashboard doesn't show SAST results. On failure, -the analyzer outputs an [exit code](../../../development/integrations/secure.md#exit-code). - -## View total number of vulnerabilities per scan - -To view the total number of vulnerabilities per scan: - -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **CI/CD > Pipelines**. -1. Select the **Status** of a branch. -1. Select the **Security** tab. - -**Scan details** shows vulnerabilities introduced by the merge request, in addition to existing vulnerabilities -from the latest successful pipeline in your project's default branch. - -### Download security scan outputs - -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3728) in GitLab 13.10. -> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/333660) in GitLab 14.2. - -Depending on the type of security scanner, you can download: - -- A JSON artifact that contains the security scanner [report](../../../development/integrations/secure.md#report). -- A CSV file that contains URLs and endpoints scanned by the security scanner. - -To download a security scan output: - -1. On the top bar, select **Menu > Projects** and find your project. -1. On the left sidebar, select **CI/CD > Pipelines**. -1. Select the **Status** of a branch. -1. Select the **Security** tab. -1. In **Scan details**, select **Download results**: - - To download a JSON file, select the JSON artifact. - - To download a CSV file, select **Download scanned resources**. - ## View vulnerabilities over time for a project > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/235558) in GitLab 13.6. diff --git a/doc/user/application_security/terminology/index.md b/doc/user/application_security/terminology/index.md index 392bfa1dde2..d50cce3b4e8 100644 --- a/doc/user/application_security/terminology/index.md +++ b/doc/user/application_security/terminology/index.md @@ -220,11 +220,12 @@ once it's imported into the database. The type of scan. This must be one of the following: +- `cluster_image_scanning` - `container_scanning` -- `dependency_scanning` - `dast` +- `dependency_scanning` - `sast` -- `cluster_image_scanning` +- `secret_detection` ### Scanner diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md index f5b1192269d..f0ac01000ef 100644 --- a/doc/user/application_security/vulnerabilities/index.md +++ b/doc/user/application_security/vulnerabilities/index.md @@ -153,7 +153,7 @@ The following scanners are supported by this feature: - [Dependency Scanning](../dependency_scanning/index.md). Automatic Patch creation is only available for Node.js projects managed with - `yarn`. + `yarn` when [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) is disabled. - [Container Scanning](../container_scanning/index.md). To resolve a vulnerability, you can either: @@ -206,7 +206,12 @@ To enable security training for vulnerabilities in your project: > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6176) in GitLab 14.9. -If security training is enabled, the vulnerability page includes a training link relevant to the detected vulnerability. +The vulnerability page may include a training link relevant to the detected vulnerability if security training is enabled. +The availability of training depends on whether the enabled training vendor has content matching the particular vulnerability. +Training content is requested based on the [vulnerability identifiers](../../../development/integrations/secure.md#identifiers). +The identifier given to a vulnerability will vary from one vulnerability to the next. The available training +content varies between vendors. This means some vulnerabilities will display no training content. +Vulnerabilities with a CWE are most likely to return a training result. To view the security training for a vulnerability: diff --git a/doc/user/application_security/vulnerability_report/pipeline.md b/doc/user/application_security/vulnerability_report/pipeline.md new file mode 100644 index 00000000000..14c13f74a5e --- /dev/null +++ b/doc/user/application_security/vulnerability_report/pipeline.md @@ -0,0 +1,141 @@ +--- +type: reference, howto +stage: Secure +group: Threat Insights +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# View vulnerabilities in a pipeline + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13496) in GitLab 12.3. + +To view vulnerabilities in a pipeline: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **CI/CD > Pipelines**. +1. From the list, select the pipeline you want to check for vulnerabilities. +1. Select the **Security** tab. + +A pipeline consists of multiple jobs, which may include security scans. When a job declares and produces security scan +reports using [`artifacts:reports`](../../../ci/yaml/artifacts_reports.md), GitLab parses and ingests the contents of +these reports to create vulnerabilities associated with the project the pipeline belongs to. + +If a job fails to finish, the pipeline vulnerability report doesn't show vulnerability findings detected by this job. +For example, if a pipeline contains DAST and SAST jobs, but the DAST job fails by returning a non-zero +[exit code](../../../development/integrations/secure.md#exit-code), the report doesn't show DAST results. + +The pipeline vulnerability report only shows results contained in the security report artifacts. This report differs from +the [Vulnerability Report](index.md), which contains cumulative results of all successful jobs, and from the merge request +[security widget](../#view-security-scan-information-in-merge-requests), which combines the branch results with +cumulative results. + +Before GitLab displays results, the vulnerability findings in all pipeline reports are [deduplicated](#deduplication-process). + +## Scan details + +**Scan details** shows a summary of vulnerability findings in the pipeline and the source reports. + +GitLab displays one row of information for each [scan type](../terminology/#scan-type-report-type) artifact present in +the pipeline. + +Note that each scan type's total number of vulnerabilities includes dismissed findings. If the number of findings +in the report doesn't match the number in **Scan details**, ensure that **Hide dismissed** is disabled. + +### Download security scan outputs + +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3728) in GitLab 13.10. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/333660) in GitLab 14.2. + +Depending on the type of security scanner, you can download: + +- A JSON artifact that contains the security scanner [report](../../../development/integrations/secure.md#report). +- A CSV file that contains URLs and endpoints scanned by the security scanner. + +To download a security scan output: + +1. In **Scan details**, select **Download results**: + - To download a JSON file, select the JSON artifact. + - To download a CSV file, select **Download scanned resources**. + +## Scan results + +This shows a list of the combined results for all security report artifacts. The filters work like the +[Vulnerability Report filters](index.md#vulnerability-report-filters), but they are limited to **Severity** and **Tool**, with +the addition of a **Hide dismissed** toggle. + +When you review the vulnerability findings reported in the pipeline, you can select one or more entries for dismissal, +similar to [Dismissing a vulnerability](index.md#dismissing-a-vulnerability) in the Vulnerability Report. + +When you merge the branch corresponding to the pipeline into the default branch, all reported findings are combined into +the [Vulnerability Report](index.md). Scan results in pipelines executed on the default branch are +incorporated once the pipeline finishes. + +| Existing vulnerability status | Dismissed in pipeline? | New vulnerability status | +|:------------------------------|:-----------------------|:-------------------------| +| any | Yes | Dismissed | +| Dismissed | any | Dismissed | +| Confirmed | No | Confirmed | +| Needs triage (Detected) | No | Needs triage (Detected) | +| Resolved | No | Needs triage (Detected) | +| N/A (i.e.: new vulnerability) | No | Needs triage (Detected) | + +## Deduplication process + +When a pipeline contains jobs that produce multiple security reports of the same type, it is possible that the same +vulnerability finding is present in multiple reports. This duplication is common when different scanners are used to +increase coverage. The deduplication process allows you to maximize the vulnerability scanning coverage while reducing +the number of findings you need to manage. + +A finding is considered a duplicate of another finding when their [scan type](../terminology/#scan-type-report-type), +[location](../terminology/#location-fingerprint) and +[identifiers](../../../development/integrations/secure.md#identifiers) are the same. + +The scan type must match because each can have its own definition for the location of a vulnerability. For example, +static analyzers are able to locate a file path and line number, whereas a container scanning analyzer uses the image +name instead. + +When comparing identifiers, GitLab does not compare `CWE` and `WASC` during deduplication because they are +"type identifiers" and are used to classify groups of vulnerabilities. Including these identifiers results in +many findings being incorrectly considered duplicates. + +In a set of duplicated findings, the first occurrence of a finding is kept and the remaining are skipped. Security +reports are processed in alphabetical file path order, and findings are processed sequentially in the order they +appear in a report. + +### Deduplication examples + +- Example 1: matching identifiers and location, mismatching scan type. + - Finding + - Scan type: `sast` + - Location fingerprint: `adc83b19e793491b1c6ea0fd8b46cd9f32e592fc` + - Identifiers: CVE-2022-25510 + - Other Finding + - Scan type: `secret_detection` + - Location fingerprint: `adc83b19e793491b1c6ea0fd8b46cd9f32e592fc` + - Identifiers: CVE-2022-25510 + - Deduplication result: not duplicates because the scan type is different. +- Example 2: matching location and scan type, mismatching type identifiers. + - Finding + - Scan type: `sast` + - Location fingerprint: `adc83b19e793491b1c6ea0fd8b46cd9f32e592fc` + - Identifiers: CWE-259 + - Other Finding + - Scan type: `sast` + - Location fingerprint: `adc83b19e793491b1c6ea0fd8b46cd9f32e592fc` + - Identifiers: CWE-798 + - Deduplication result: duplicates because `CWE` identifiers are ignored. +- Example 3: matching scan type, location and identifiers. + - Finding + - Scan type: `container_scanning` + - Location fingerprint: `adc83b19e793491b1c6ea0fd8b46cd9f32e592fc` + - Identifiers: CVE-2022-25510, CWE-259 + - Other Finding + - Scan type: `container_scanning` + - Location fingerprint: `adc83b19e793491b1c6ea0fd8b46cd9f32e592fc` + - Identifiers: CVE-2022-25510, CWE-798 + - Deduplication result: duplicates because all criteria match, and type identifiers are ignored. + +The examples above don't include the raw location values. Each scan type defines its own +`fingerprint_data`, which is used to generate a `SHA1` hash that is used as the `location_fingerprint`. +You can find definitions for each scan type [`gitlab/lib/gitlab/ci/reports/security/locations`](https://gitlab.com/gitlab-org/gitlab/-/tree/01c69e97340b7c1c7e30c0caec8506910b6503c8/lib/gitlab/ci/reports/security/locations) +and [`gitlab/ee/lib/gitlab/ci/reports/security/locations`](https://gitlab.com/gitlab-org/gitlab/-/tree/01c69e97340b7c1c7e30c0caec8506910b6503c8/ee/lib/gitlab/ci/reports/security/locations). diff --git a/doc/user/clusters/agent/ci_cd_workflow.md b/doc/user/clusters/agent/ci_cd_workflow.md index c04c5a1f7ec..dce02a72300 100644 --- a/doc/user/clusters/agent/ci_cd_workflow.md +++ b/doc/user/clusters/agent/ci_cd_workflow.md @@ -42,17 +42,19 @@ If you have multiple GitLab projects that contain Kubernetes manifests: 1. [Install the GitLab agent](install/index.md) in its own project, or in one of the GitLab projects where you keep Kubernetes manifests. 1. [Authorize the agent](#authorize-the-agent) to access your GitLab projects. -1. Optional. For added security, [use impersonation](#use-impersonation-to-restrict-project-and-group-access). +1. Optional. For added security, [use impersonation](#restrict-project-and-group-access-by-using-impersonation). 1. [Update your `.gitlab-ci.yml` file](#update-your-gitlab-ciyml-file-to-run-kubectl-commands) to select the agent's Kubernetes context and run the Kubernetes API commands. 1. Run your pipeline to deploy to or update the cluster. ## Authorize the agent -You must authorize the agent to access the project where you keep your Kubernetes manifests. +If you have multiple GitLab projects, you must authorize the agent to access the project where you keep your Kubernetes manifests. You can authorize the agent to access individual projects, or authorize a group or subgroup, so all projects within have access. For added security, you can also -[use impersonation](#use-impersonation-to-restrict-project-and-group-access). +[use impersonation](#restrict-project-and-group-access-by-using-impersonation). + +Authorization configuration can take one or two minutes to propagate. ### Authorize the agent to access your projects @@ -60,7 +62,7 @@ so all projects within have access. For added security, you can also To authorize the agent to access the GitLab project where you keep Kubernetes manifests: -1. On the top bar, select **Menu > Projects** and find the project that contains the agent configuration file (`config.yaml`). +1. On the top bar, select **Menu > Projects** and find the project that contains the [agent configuration file](install/index.md#create-an-agent-configuration-file) (`config.yaml`). 1. Edit the `config.yaml` file. Under the `ci_access` keyword, add the `projects` attribute. 1. For the `id`, add the path: @@ -83,7 +85,7 @@ Choose the context to run `kubectl` commands from your CI/CD scripts. To authorize the agent to access all of the GitLab projects in a group or subgroup: -1. On the top bar, select **Menu > Projects** and find the project that contains the agent configuration file (`config.yaml`). +1. On the top bar, select **Menu > Projects** and find the project that contains the [agent configuration file](install/index.md#create-an-agent-configuration-file) (`config.yaml`). 1. Edit the `config.yaml` file. Under the `ci_access` keyword, add the `groups` attribute. 1. For the `id`, add the path: @@ -154,7 +156,7 @@ deploy: # ... rest of your job configuration ``` -## Use impersonation to restrict project and group access **(PREMIUM)** +## Restrict project and group access by using impersonation **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345014) in GitLab 14.5. @@ -170,6 +172,8 @@ You can impersonate: - The CI/CD job that accesses the cluster. - A specific user or system account defined within the cluster. +Authorization configuration can take one or two minutes to propagate. + ### Impersonate the agent The agent is impersonated by default. You don't need to do anything to impersonate it. @@ -220,6 +224,24 @@ ci_access: ci_job: {} ``` +#### Example RBAC to restrict CI/CD jobs + +The following `RoleBinding` resource restricts all CI/CD jobs to view rights only. + +```yaml +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: ci-job-view +roleRef: + name: view + kind: ClusterRole + apiGroup: rbac.authorization.k8s.io +subjects: + - name: gitlab:ci_job + kind: Group +``` + ### Impersonate a static identity For a given connection, you can use a static identity for the impersonation. diff --git a/doc/user/clusters/agent/gitops.md b/doc/user/clusters/agent/gitops.md index 64eae308bec..73a35ffbc64 100644 --- a/doc/user/clusters/agent/gitops.md +++ b/doc/user/clusters/agent/gitops.md @@ -57,7 +57,7 @@ Any time you commit updates to your Kubernetes manifests, the agent updates the ## GitOps configuration reference -The following snippet shows an example of the possible keys and values for the GitOps section of an agent configuration file. +The following snippet shows an example of the possible keys and values for the GitOps section of an [agent configuration file](install/index.md#create-an-agent-configuration-file) (`config.yaml`). ```yaml gitops: diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index 5a69da28632..0d2b68e154d 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -43,21 +43,17 @@ This workflow is considered push-based, because GitLab is pushing requests from GitLab supports the following Kubernetes versions. You can upgrade your Kubernetes version to a supported version at any time: -- 1.22 (support ends on March 22, 2023) -- 1.21 (support ends on November 22, 2022) -- 1.20 (support ends on July 22, 2022) +- 1.24 (support ends on September 22, 2023 or when 1.27 becomes supported) +- 1.23 (support ends on February 22, 2023 or when 1.26 becomes supported) +- 1.22 (support ends on October 22, 2022) +- 1.21 (support ends on August 22, 2022) -GitLab supports at least two production-ready Kubernetes minor -versions at any given time. GitLab regularly reviews the supported versions and -provides a three-month deprecation period before removing support for a specific -version. The list of supported versions is based on: +GitLab aims to support a new minor Kubernetes version three months after its initial release. GitLab supports at least three production-ready Kubernetes minor +versions at any given time. -- The versions supported by major managed Kubernetes providers. -- The versions [supported by the Kubernetes community](https://kubernetes.io/releases/version-skew-policy/#supported-versions). +Support for deprecated APIs can be removed from the GitLab codebase when we drop support for the Kubernetes version that only supports the deprecated API. -[This epic](https://gitlab.com/groups/gitlab-org/-/epics/4827) tracks support for other Kubernetes versions. - -Some GitLab features might work on versions not listed here. +Some GitLab features might work on versions not listed here. [This epic](https://gitlab.com/groups/gitlab-org/-/epics/4827) tracks support for Kubernetes versions. ## Migrate to the agent from the legacy certificate-based integration diff --git a/doc/user/clusters/agent/install/index.md b/doc/user/clusters/agent/install/index.md index 6c839f5ffc6..9282ac7fb40 100644 --- a/doc/user/clusters/agent/install/index.md +++ b/doc/user/clusters/agent/install/index.md @@ -89,8 +89,12 @@ You must register an agent before you can install the agent in your cluster. To - If you want to create a configuration with CI/CD defaults, type a name. - If you already have an [agent configuration file](#create-an-agent-configuration-file), select it from the list. 1. Select **Register an agent**. -1. GitLab generates an access token for the agent. Securely store this token. You need it to install the agent +1. GitLab generates an access token for the agent. You need this token to install the agent in your cluster and to [update the agent](#update-the-agent-version) to another version. + + WARNING: + Securely store the agent access token. A bad actor can use this token to access source code in the agent's configuration project, access source code in any public project on the GitLab instance, or even, under very specific conditions, obtain a Kubernetes manifest. + 1. Copy the command under **Recommended installation method**. You need it when you use the one-liner installation method to install the agent in your cluster. @@ -154,8 +158,9 @@ GitLab also provides a [KPT package for the agent](https://gitlab.com/gitlab-org To configure your agent, add content to the `config.yaml` file: -- [View the configuration reference](../gitops.md#gitops-configuration-reference) for a GitOps workflow. -- [View the configuration reference](../ci_cd_workflow.md) for a GitLab CI/CD workflow. +- For a GitOps workflow, [view the configuration reference](../gitops.md#gitops-configuration-reference). +- For a GitLab CI/CD workflow, [authorize the agent to access your projects](../ci_cd_workflow.md#authorize-the-agent). Then + [add `kubectl` commands to your `.gitlab-ci.yml` file](../ci_cd_workflow.md#update-your-gitlab-ciyml-file-to-run-kubectl-commands). ## Install multiple agents in your cluster diff --git a/doc/user/clusters/agent/troubleshooting.md b/doc/user/clusters/agent/troubleshooting.md index 0932e9179f9..0596755ec74 100644 --- a/doc/user/clusters/agent/troubleshooting.md +++ b/doc/user/clusters/agent/troubleshooting.md @@ -186,3 +186,25 @@ Alternatively, you can mount the certificate file at a different location and sp This error occurs when the project where you keep your manifests is not public. To fix it, make sure your project is public or your manifest files are stored in the repository where the agent is configured. + +## Failed to perform vulnerability scan on workload: jobs.batch already exists + +```json +{ + "level": "error", + "time": "2022-06-22T21:03:04.769Z", + "msg": "Failed to perform vulnerability scan on workload", + "mod_name": "starboard_vulnerability", + "error": "running scan job: creating job: jobs.batch \"scan-vulnerabilityreport-b8d497769\" already exists" +} +``` + +The GitLab agent performs vulnerability scans by creating a job to scan each workload. If a scan +is interrupted, these jobs may be left behind and will need to be cleaned up before more jobs can +be run. You can clean up these jobs by running: + +```shell +kubectl delete jobs -l app.kubernetes.io/managed-by=starboard -n gitlab-agent +``` + +[We're working on making the cleanup of these jobs more robust.](https://gitlab.com/gitlab-org/gitlab/-/issues/362016) diff --git a/doc/user/clusters/agent/vulnerabilities.md b/doc/user/clusters/agent/vulnerabilities.md index 706ed122f7b..3b80a7a0f81 100644 --- a/doc/user/clusters/agent/vulnerabilities.md +++ b/doc/user/clusters/agent/vulnerabilities.md @@ -11,26 +11,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w To view cluster vulnerabilities, you can view the [vulnerability report](../../application_security/vulnerabilities/index.md). You can also configure your agent so the vulnerabilities are displayed with other agent information in GitLab. -## View cluster vulnerabilities - -Prerequisite: - -- You must have at least the Developer role. - -To view vulnerability information in GitLab: - -1. On the top bar, select **Menu > Projects** and find the project that contains the agent configuration file. -1. On the left sidebar, select **Infrastructure > Kubernetes clusters**. -1. Select the **Agent** tab. -1. Select the agent you want to see the vulnerabilities for. - -![Cluster agent security tab UI](../img/cluster_agent_security_tab_v14_8.png) - -This information can also be found under [operational vulnerabilities](../../../user/application_security/vulnerability_report/index.md#operational-vulnerabilities). - -## Enable cluster vulnerability scanning **(ULTIMATE)** +## Enable operational container scanning **(ULTIMATE)** -You can use [cluster image scanning](../../application_security/cluster_image_scanning/index.md) +You can use operational container scanning to scan container images in your cluster for security vulnerabilities. To begin scanning all resources in your cluster, add a `starboard` @@ -49,7 +32,7 @@ The `cadence` field is required. GitLab supports the following types of CRON syn It is possible that other elements of the CRON syntax will work in the cadence field, however, GitLab does not officially test or support them. -By default, cluster image scanning will attempt to scan the workloads in all +By default, operational container scanning will attempt to scan the workloads in all namespaces for vulnerabilities. The `vulnerability_report` block has a `namespaces` field which can be used to restrict which namespaces are scanned. For example, if you would like to scan only the `development`, `staging`, and `production` @@ -64,3 +47,20 @@ starboard: - staging - production ``` + +## View cluster vulnerabilities + +Prerequisite: + +- You must have at least the Developer role. + +To view vulnerability information in GitLab: + +1. On the top bar, select **Menu > Projects** and find the project that contains the agent configuration file. +1. On the left sidebar, select **Infrastructure > Kubernetes clusters**. +1. Select the **Agent** tab. +1. Select an agent to view the cluster vulnerabilities. + +![Cluster agent security tab UI](../img/cluster_agent_security_tab_v14_8.png) + +This information can also be found under [operational vulnerabilities](../../../user/application_security/vulnerability_report/index.md#operational-vulnerabilities). diff --git a/doc/user/clusters/agent/work_with_agent.md b/doc/user/clusters/agent/work_with_agent.md index 8872ecf7ce5..058243ec218 100644 --- a/doc/user/clusters/agent/work_with_agent.md +++ b/doc/user/clusters/agent/work_with_agent.md @@ -53,24 +53,40 @@ View and provide feedback about the UI in [this epic](https://gitlab.com/groups/ ## Debug the agent +> The `grpc_level` was [introduced](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/merge_requests/669) in GitLab 15.1. + To debug the cluster-side component (`agentk`) of the agent, set the log level according to the available options: -- `off` -- `warning` - `error` +- `warning` - `info` - `debug` -The log level defaults to `info`. You can change it by using a top-level `observability` -section in the configuration file, for example: +The agent has two loggers: + +- A general purpose logger, which defaults to `info`. +- A gRPC logger, which defaults to `error`. + +One can change their log levels by using a top-level `observability` section in the [agent configuration file](install/index.md#configure-your-agent), for example setting the levels to `debug` and `warning`: ```yaml observability: logging: level: debug + grpc_level: warning ``` +When `grpc_level` is set to `info` or below, there will be a lot of gRPC logs. + +Commit the configuration changes and inspect the agent service logs: + +```shell +kubectl logs -f -l=app=gitlab-agent -n gitlab-agent +``` + +For more information about debugging, see [troubleshooting documentation](troubleshooting.md). + ## Reset the agent token > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327152) in GitLab 14.9. diff --git a/doc/user/clusters/integrations.md b/doc/user/clusters/integrations.md index 94fb443e0fb..c7597896575 100644 --- a/doc/user/clusters/integrations.md +++ b/doc/user/clusters/integrations.md @@ -28,13 +28,11 @@ You can install your applications manually as shown in the following sections, o installation. Although, the [Cluster management project template](management_project_template.md) still -requires that you manually do the last steps of these sections, -[Enable Prometheus integration for your cluster](#enable-prometheus-integration-for-your-cluster) -or [Enable Elastic Stack integration for your cluster](#enable-elastic-stack-integration-for-your-cluster) -depending on which application you are installing. [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/326565) +requires that you manually do the last steps of this section, +[Enable Prometheus integration for your cluster](#enable-prometheus-integration-for-your-cluster). [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/326565) to automate this step. -Prometheus and Elastic Stack cluster integrations can only be enabled for clusters [connected through cluster certificates](../project/clusters/add_existing_cluster.md). +Prometheus cluster integrations can only be enabled for clusters [connected through cluster certificates](../project/clusters/add_existing_cluster.md). To enable Prometheus for your cluster connected through the [GitLab agent](agent/index.md), you can [integrate it manually](../project/integrations/prometheus.md#manual-configuration-of-prometheus). @@ -100,70 +98,3 @@ To enable the Prometheus integration for your cluster: 1. Check the **Enable Prometheus integration** checkbox. 1. Select **Save changes**. 1. Go to the **Health** tab to see your cluster's metrics. - -## Elastic Stack cluster integration **(FREE SELF)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/61077) in GitLab 13.12. -> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346485) in GitLab 14.7. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/360182) behind a [feature flag](../../administration/feature_flags.md) named `monitor_logging` in GitLab 15.0. Disabled by default. - -WARNING: -This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/346485) -in GitLab 14.7. -It will be removed completely in GitLab 15.2. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `monitor_logging`. -On GitLab.com, this feature is not available. -This feature is not recommended for production use. - -You can integrate your cluster with [Elastic -Stack](https://www.elastic.co/elastic-stack/) to index and [query your pod -logs](../project/clusters/kubernetes_pod_logs.md). - -### Elastic Stack Prerequisites - -To use this integration: - -1. Elasticsearch 7.x or must be installed in your cluster in the - `gitlab-managed-apps` namespace. -1. The `Service` resource must be called `elastic-stack-elasticsearch-master` - and expose the Elasticsearch API on port `9200`. -1. The logs are expected to be [Filebeat container logs](https://www.elastic.co/guide/en/beats/filebeat/7.16/filebeat-input-container.html) - following the [7.x log structure](https://www.elastic.co/guide/en/beats/filebeat/7.16/exported-fields-log.html) - and include [Kubernetes metadata](https://www.elastic.co/guide/en/beats/filebeat/7.16/add-kubernetes-metadata.html). - -You can manage your Elastic Stack however you like, but as an example, you can -use [this Elastic Stack chart](https://gitlab.com/gitlab-org/charts/elastic-stack) to get up and -running: - -```shell -# Create the required Kubernetes namespace -kubectl create namespace gitlab-managed-apps - -# Download Helm chart values that is compatible with the requirements above. -# These are included in the Cluster Management project template. -wget https://gitlab.com/gitlab-org/project-templates/cluster-management/-/raw/master/applications/elastic-stack/values.yaml - -# Add the GitLab Helm chart repository -helm repo add gitlab https://charts.gitlab.io - -# Install Elastic Stack -helm install elastic-stack gitlab/elastic-stack -n gitlab-managed-apps --values values.yaml -``` - -### Enable Elastic Stack integration for your cluster - -To enable the Elastic Stack integration for your cluster: - -1. Go to the cluster's page: - - For a [project-level cluster](../project/clusters/index.md), navigate to your project's - **Infrastructure > Kubernetes clusters**. - - For a [group-level cluster](../group/clusters/index.md), navigate to your group's - **Kubernetes** page. - - For an [instance-level cluster](../instance/clusters/index.md), navigate to your instance's - **Kubernetes** page. -1. Select the **Integrations** tab. -1. Check the **Enable Elastic Stack integration** checkbox. -1. Select **Save changes**. -1. Go to the **Health** tab to see your cluster's metrics. diff --git a/doc/user/compliance/compliance_report/index.md b/doc/user/compliance/compliance_report/index.md index 0006ae02752..f547e5f146f 100644 --- a/doc/user/compliance/compliance_report/index.md +++ b/doc/user/compliance/compliance_report/index.md @@ -96,7 +96,7 @@ Our criteria for the separation of duties is as follows: The Chain of Custody report allows customers to export a list of merge commits within the group. The data provides a comprehensive view with respect to merge commits. It includes the merge commit SHA, -merge request author, merge request ID, merge user, pipeline ID, group name, project name, and merge request approvers. +merge request author, merge request ID, merge user, date merged, pipeline ID, group name, project name, and merge request approvers. Depending on the merge strategy, the merge commit SHA can be a merge commit, squash commit, or a diff head commit. To download the Chain of Custody report: diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index 659c0326728..8c57220068b 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -35,8 +35,7 @@ compliance report is shown properly. The results are saved as a [License Compliance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportslicense_scanning) -that you can later download and analyze. Due to implementation limitations, we -always take the latest License Compliance artifact available. +that you can later download and analyze. WARNING: License Compliance Scanning does not support run-time installation of compilers and interpreters. @@ -72,11 +71,13 @@ Gradle 1.x projects are not supported. The minimum supported version of Maven is |------------|----------------------------------------------------------------------------------------------|-------| | JavaScript | [Bower](https://bower.io/), [npm](https://www.npmjs.com/) (7 and earlier) | | | Go | [Godep](https://github.com/tools/godep) ([deprecated](../../../update/deprecations.md#godep-support-in-license-compliance)), [go mod](https://github.com/golang/go/wiki/Modules) | | -| Java | [Gradle](https://gradle.org/), [Maven](https://maven.apache.org/) | | +| Java | [Gradle](https://gradle.org/) <sup>1</sup>, [Maven](https://maven.apache.org/) | | | .NET | [NuGet](https://www.nuget.org/) | The .NET Framework is supported via the [mono project](https://www.mono-project.com/). There are, however, some limitations. The scanner doesn't support Windows-specific dependencies and doesn't report dependencies of your project's listed dependencies. Also, the scanner always marks detected licenses for all dependencies as `unknown`. | | Python | [pip](https://pip.pypa.io/en/stable/) | Python is supported through [requirements.txt](https://pip.pypa.io/en/stable/user_guide/#requirements-files) and [Pipfile.lock](https://github.com/pypa/pipfile#pipfilelock). | | Ruby | [gem](https://rubygems.org/) | | +1. Gradle 7 and later is not supported as dependencies are not discovered when included with the `implementation` directive. Please see [GitLab#341222](https://gitlab.com/gitlab-org/gitlab/-/issues/341222) for more details. + ### Experimental support The following languages and package managers are [supported experimentally](https://github.com/pivotal/LicenseFinder#experimental-project-types). @@ -101,6 +102,8 @@ To enable License Compliance in your project's pipeline, either: (provided by [Auto DevOps](../../../topics/autodevops/index.md)). - Include the [`License-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/License-Scanning.gitlab-ci.yml) in your `.gitlab-ci.yml` file. +Please note that License Compliance is not supported when GitLab is run with FIPS mode enabled. + ### Include the License Scanning template Prerequisites: @@ -110,6 +113,7 @@ Prerequisites: shared runners on GitLab.com, this is enabled by default. - License Scanning runs in the `test` stage, which is available by default. If you redefine the stages in the `.gitlab-ci.yml` file, the `test` stage is required. +- [FIPS mode](../../../development/fips_compliance.md#enable-fips-mode) must be disabled. To [include](../../../ci/yaml/index.md#includetemplate) the [`License-Scanning.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/License-Scanning.gitlab-ci.yml), add it to your `.gitlab-ci.yml` file: diff --git a/doc/user/crm/index.md b/doc/user/crm/index.md index b5287816052..a2cfdf61a8d 100644 --- a/doc/user/crm/index.md +++ b/doc/user/crm/index.md @@ -39,7 +39,7 @@ To enable customer relations management in a group or subgroup: 1. On the top bar, select **Menu > Groups** and find your group or subgroup. 1. On the left sidebar, select **Settings > General**. 1. Expand the **Permissions and group features** section. -1. Select **Enable customer relations**. +1. Select **Customer relations is enabled**. 1. Select **Save changes**. ## Contacts @@ -118,6 +118,9 @@ organizations using the GraphQL API. ## Issues +If you use [Service Desk](../project/service_desk.md) and create issues from emails, +issues are linked to contacts matching the email addresses in the sender and CC of the email. + ### View issues linked to a contact To view a contact's issues, select a contact from the issue sidebar, or: @@ -170,10 +173,7 @@ API. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2256) in GitLab 14.8 [with a flag](../../administration/feature_flags.md) named `contacts_autocomplete`. Disabled by default. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/352123) in GitLab 15.0. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `contacts_autocomplete`. -On GitLab.com, this feature is available. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352123) in GitLab 15.2. [Feature flag `contacts_autocomplete`](https://gitlab.com/gitlab-org/gitlab/-/issues/352123) removed. When you use the `/add_contacts` or `/remove_contacts` quick actions, follow them with `[contact:` and an autocomplete list appears: diff --git a/doc/user/discussions/img/start_image_discussion.gif b/doc/user/discussions/img/start_image_discussion.gif Binary files differdeleted file mode 100644 index 18b2a4701cc..00000000000 --- a/doc/user/discussions/img/start_image_discussion.gif +++ /dev/null diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index a0649a61905..1f34d182718 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -98,8 +98,6 @@ This comment can also be a thread. An icon is displayed on the image and a comment field is displayed. -![Start image thread](img/start_image_discussion.gif) - ## Reply to a comment by sending email If you have ["reply by email"](../../administration/reply_by_email.md) configured, @@ -156,11 +154,7 @@ If an issue or merge request is locked and closed, you cannot reopen it. > - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/351143) in GitLab 14.10: you can only mark comments in issues and epics as confidential. Previously, it was also possible for comments in merge requests and snippets. > - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87403) from "confidential comments" to "internal notes" in GitLab 15.0. > - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/87383) in GitLab 15.0. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, -ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `confidential_notes`. -On GitLab.com, this feature is available. +> - [Feature flag `confidential_notes`](https://gitlab.com/gitlab-org/gitlab/-/issues/362712) removed in GitLab 15.2. You can add an internal note **to an issue or an epic**. It's then visible only to the following people: @@ -229,7 +223,7 @@ To change the activity sort order: ## Assign an issue to the commenting user -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/191455) in GitLab 13.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/191455) in GitLab 13.1. You can assign an issue to a user who made a comment. diff --git a/doc/user/free_user_limit.md b/doc/user/free_user_limit.md index 868e322cac9..b848128b160 100644 --- a/doc/user/free_user_limit.md +++ b/doc/user/free_user_limit.md @@ -1,6 +1,6 @@ --- stage: Growth -group: Conversion +group: Acquisition info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index d515f9f4558..529b81e2645 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -230,11 +230,23 @@ also load certain page content directly from common public CDN hostnames. ## Webhooks -The following limits apply for [webhooks](../project/integrations/webhooks.md): +The following limits apply for [webhooks](../project/integrations/webhooks.md). + +### Rate limits + +The number of times a webhook can be called per minute, per top-level namespace. +The limit varies depending on your plan and the number of seats in your subscription. + +| Plan | Default for GitLab.com | +|----------------------|-------------------------| +| Free | `500` | +| Premium | `99` seats or fewer: `1,600`<br>`100-399` seats: `2,800`<br>`400` seats or more: `4,000` | +| Ultimate and open source |`999` seats or fewer: `6,000`<br>`1,000-4,999` seats: `9,000`<br>`5,000` seats or more: `13,000` | + +### Other limits | Setting | Default for GitLab.com | |----------------------|-------------------------| -| Webhook rate limit | `500` calls per minute for GitLab Free, unlimited for GitLab Premium and GitLab Ultimate. Webhook rate limits are applied per top-level namespace. | | Number of webhooks | `100` per project, `50` per group | | Maximum payload size | 25 MB | @@ -344,6 +356,7 @@ after the limits change in January, 2021: | **GitLab Pages** requests (for a given **IP address**) | | **1000** requests per **50 seconds** | | **GitLab Pages** requests (for a given **GitLab Pages domain**) | | **5000** requests per **10 seconds** | | **Pipeline creation** requests (for a given **project, user, and commit**) | | **25** requests per minute | +| **Alert integration endpoint** requests (for a given **project**) | | **3600** requests per hour | More details are available on the rate limits for [protected paths](#protected-paths-throttle) and [raw @@ -407,12 +420,7 @@ No response headers are provided. ### Pagination response headers -For performance reasons, if a query returns more than 10,000 records, GitLab -doesn't return the following headers: - -- `x-total`. -- `x-total-pages`. -- `rel="last"` `link`. +For performance reasons, if a query returns more than 10,000 records, [GitLab excludes some headers](../../api/index.md#pagination-response-headers). ### Visibility settings diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index e0334eda875..71d7b7fbb0c 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -34,6 +34,7 @@ To create an epic in the group you're in: - To [make the epic confidential](#make-an-epic-confidential), select the checkbox under **Confidentiality**. - Choose labels. - Select a start and due date, or [inherit](#start-and-due-date-inheritance) them. + - Select a [color](#epic-color). 1. Select **Create epic**. The newly created epic opens. @@ -62,6 +63,18 @@ Because the epic's dates can inherit dates from its children, the start date and If the start date of a child epic on the lowest level changes, that becomes the earliest possible start date for its parent epic. The parent epic's start date then reflects this change and propagates upwards to the top epic. +### Epic color + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/79940) in GitLab 14.9 [with a flag](../../../administration/feature_flags.md) named `epic_color_highlight`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `epic_color_highlight`. +On GitLab.com, this feature is available but can be configured by GitLab.com administrators only. +The feature is not ready for production use. + +When you create or edit an epic, you can select its color. +An epic's color is shown in [roadmaps](../roadmap/index.md), and [epic boards](epic_boards.md). + ## Edit an epic After you create an epic, you can edit the following details: @@ -71,6 +84,7 @@ After you create an epic, you can edit the following details: - Start date - Due date - Labels +- [Color](#epic-color) Prerequisites: diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index ae1465d0b1b..edf4d7677df 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -86,7 +86,7 @@ migrated: - Badges ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292431) in 13.11) - Board Lists -- Boards +- Boards - Epics ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250281) in 13.7) - Finisher - Group Labels ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292429) in 13.9) diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 6ba8251ba05..56d1569c908 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -45,16 +45,19 @@ the immediate parent group. ### Namespaces -In GitLab, a namespace is a unique name for a user, a group, or subgroup under -which a project can be created. +In GitLab, a *namespace* organizes related projects together. +GitLab has two types of namespaces: -For example, consider a user named Alex: +- A *personal* namespace, which is based on your username. Projects under a personal namespace must be configured one at a time. +- A *group* or *subgroup* namespace. In these namespaces, you can manage multiple projects at once. -| GitLab URL | Namespace | -| ---------- | --------- | -| Alex creates an account with the username `alex`: `https://gitlab.example.com/alex`. | The namespace in this case is `alex`. | -| Alex creates a group for their team with the group name `alex-team`. The group and its projects are available at: `https://gitlab.example.com/alex-team`. | The namespace in this case is `alex-team`. | -| Alex creates a subgroup of `alex-team` with the subgroup name `marketing`. The subgroup and its projects are available at: `https://gitlab.example.com/alex-team/marketing`. | The namespace in this case is `alex-team/marketing`. | +To determine whether you're viewing a group or personal namespace, you can view the URL. For example: + +| Namespace for | URL | Namespace | +| ------------- | --- | --------- | +| A user named `alex`. | `https://gitlab.example.com/alex` | `alex` | +| A group named `alex-team`. | `https://gitlab.example.com/alex-team` | `alex-team` | +| A group named `alex-team` with a subgroup named `marketing`. | `https://gitlab.example.com/alex-team/marketing` | `alex-team/marketing` | ## Create a group @@ -240,7 +243,7 @@ To change this setting for a specific group: 1. Find the group and select it. 1. From the left menu, select **Settings > General**. 1. Expand the **Permissions and group features** section. -1. Select the desired option in the **Allowed to create projects** dropdown list. +1. Select the desired option in the **Roles allowed to create projects** dropdown list. 1. Select **Save changes**. To change this setting globally, see [Default project creation protection](../admin_area/settings/visibility_and_access_controls.md#define-which-roles-can-create-projects). @@ -478,7 +481,7 @@ To prevent sharing outside of the group's hierarchy: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Settings > General**. 1. Expand **Permissions and group features**. -1. Select **Prevent members from sending invitations to groups outside of `<group_name>` and its subgroups**. +1. Select **Members cannot invite groups outside of `<group_name>` and its subgroups**. 1. Select **Save changes**. ## Prevent a project from being shared with groups @@ -490,7 +493,7 @@ To prevent a project from being shared with other groups: 1. Go to the group's **Settings > General** page. 1. Expand the **Permissions and group features** section. -1. Select **Prevent sharing a project in `<group_name>` with other groups**. +1. Select **Projects in `<group_name>` cannot be shared with other groups**. 1. Select **Save changes**. This setting applies to all subgroups unless overridden by a group owner. Groups already @@ -582,7 +585,7 @@ To prevent members from being added to projects in a group: 1. Go to the group's **Settings > General** page. 1. Expand the **Permissions and group features** section. -1. Under **Membership**, select **Prevent adding new members to projects within this group**. +1. Under **Membership**, select **Users cannot be added to projects in this group**. 1. Select **Save changes**. All users who previously had permissions can no longer add members to a group. @@ -608,15 +611,14 @@ To ensure only people from your organization can access particular resources, you can restrict access to groups by IP address. This group-level setting applies to: -- The GitLab UI, including subgroups, projects, and issues. +- The GitLab UI, including subgroups, projects, issues, and pages. - [In GitLab 12.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/12874), the API. +- Using Git over SSH on GitLab.com. ### Security implications You should consider some security implications before configuring IP address restrictions. -- Restricting HTTP traffic on GitLab.com with IP address restrictions causes SSH requests (including Git operations over - SSH) to fail. For more information, see [the relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/271673). - Administrators and group owners can access group settings from any IP address, regardless of IP restriction. However: - Groups owners cannot access projects belonging to the group when accessing from a disallowed IP address. - Administrators can access projects belonging to the group when accessing from a disallowed IP address. @@ -629,6 +631,8 @@ You should consider some security implications before configuring IP address res restricted IP address, the IP restriction prevents code from being cloned. - Users may still see some events from the IP restricted groups and projects on their dashboard. Activity may include push, merge, issue, or comment events. +- IP access restrictions for Git operations via SSH are supported only on GitLab SaaS. + IP access restrictions applied to self-managed instances block SSH completely. ### Restrict group access by IP address @@ -636,7 +640,7 @@ To restrict group access by IP address: 1. Go to the group's **Settings > General** page. 1. Expand the **Permissions and group features** section. -1. In the **Allow access to the following IP addresses** field, enter IPv4 or IPv6 address ranges in CIDR notation. +1. In the **Restrict access by IP address** field, enter IPv4 or IPv6 address ranges in CIDR notation. 1. Select **Save changes**. In self-managed installations of GitLab 15.1 and later, you can also configure @@ -671,6 +675,26 @@ The most popular public email domains cannot be restricted, such as: When you share a group, both the source and target namespaces must allow the domains of the members' email addresses. +## Restrict Git access protocols + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/365601) in GitLab 15.1 [with a flag](../../administration/feature_flags.md) named `group_level_git_protocol_control`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to +[enable the feature flag](../../administration/feature_flags.md) named `group_level_git_protocol_control`. On GitLab.com, +this feature is available. + +You can set the permitted protocols used to access a group's repositories to either SSH, HTTPS, or both. This setting +is disabled when the [instance setting](../admin_area/settings/visibility_and_access_controls.md#configure-enabled-git-access-protocols) is +configured by an administrator. + +To change the permitted Git access protocols for a group: + +1. Go to the group's **Settings > General** page. +1. Expand the **Permissions and group features** section. +1. Choose the permitted protocols from **Enabled Git access protocols**. +1. Select **Save changes**. + ## Group file templates **(PREMIUM)** Use group file templates to share a set of templates for common file @@ -712,7 +736,7 @@ To disable email notifications: 1. Go to the group's **Settings > General** page. 1. Expand the **Permissions and group features** section. -1. Select **Disable email notifications**. +1. Select **Email notifications are disabled**. 1. Select **Save changes**. ## Disable group mentions @@ -731,7 +755,7 @@ To disable group mentions: 1. Go to the group's **Settings > General** page. 1. Expand the **Permissions and group features** section. -1. Select **Disable group mentions**. +1. Select **Group mentions are disabled**. 1. Select **Save changes**. ## Enable delayed project deletion **(PREMIUM)** @@ -743,7 +767,7 @@ To disable group mentions: > - [User interface changed](https://gitlab.com/gitlab-org/gitlab/-/issues/352961) in GitLab 15.1. [Delayed project deletion](../project/settings/index.md#delayed-project-deletion) is locked and disabled unless the instance-level settings for -[deletion protection](../admin_area/settings/visibility_and_access_controls.md#deletion-protection) is enabled for either groups only or groups and projects. +[deletion protection](../admin_area/settings/visibility_and_access_controls.md#deletion-protection) are enabled for either groups only or groups and projects. When enabled on groups, projects in the group are deleted after a period of delay. During this period, projects are in a read-only state and can be restored. The default period is seven days but [is configurable at the instance level](../admin_area/settings/visibility_and_access_controls.md#retention-period). @@ -848,12 +872,12 @@ Support for group-level settings for merge request approval rules is tracked in - [Audit Events](../../administration/audit_events.md#group-events). - [CI/CD minutes quota](../../ci/pipelines/cicd_minutes.md): Keep track of the CI/CD minute quota for the group. - [Integrations](../admin_area/settings/project_integration_management.md). -- [Transfer a project into a group](../project/settings/index.md#transferring-an-existing-project-into-another-namespace). +- [Transfer a project into a group](../project/settings/index.md#transfer-a-project-to-another-namespace). - [Share a project with a group](../project/members/share_project_with_groups.md): Give all group members access to the project at once. - [Lock the sharing with group feature](#prevent-a-project-from-being-shared-with-groups). - [Enforce two-factor authentication (2FA)](../../security/two_factor_authentication.md#enforce-2fa-for-all-users-in-a-group): Enforce 2FA for all group members. -- Namespaces [API](../../api/namespaces.md) and [Rake tasks](../../raketasks/features.md). +- Namespaces [API](../../api/namespaces.md) and [Rake tasks](../../raketasks/index.md). - [Control access and visibility](../admin_area/settings/visibility_and_access_controls.md). ## Troubleshooting diff --git a/doc/user/group/saml_sso/group_managed_accounts.md b/doc/user/group/saml_sso/group_managed_accounts.md deleted file mode 100644 index 0a00d0c1c1c..00000000000 --- a/doc/user/group/saml_sso/group_managed_accounts.md +++ /dev/null @@ -1,14 +0,0 @@ ---- -type: reference, howto -stage: Manage -group: Authentication and Authorization -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -remove_date: '2022-06-13' -redirect_to: 'index.md' ---- - -# Group Managed Accounts **(PREMIUM)** - -This [closed beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#sts=Closed%20Beta) feature was never enabled globally. See -[this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/296544) for progress on removing the feature. -Use [SAML SSO](index.md) instead. diff --git a/doc/user/group/saml_sso/group_sync.md b/doc/user/group/saml_sso/group_sync.md index 2239562b831..b8b7a16b31b 100644 --- a/doc/user/group/saml_sso/group_sync.md +++ b/doc/user/group/saml_sso/group_sync.md @@ -23,10 +23,12 @@ For a demo of Group Sync using Azure, see [Demo: SAML Group Sync](https://youtu. To configure SAML Group Sync: -1. Configure SAML authentication: - - For GitLab self-managed, see [SAML OmniAuth Provider](../../../integration/saml.md). - - For GitLab.com, see [SAML SSO for GitLab.com groups](index.md). -1. Ensure your SAML identity provider sends an attribute statement named `Groups` or `groups`. +- For GitLab self-managed: + 1. Configure the [SAML OmniAuth Provider](../../../integration/saml.md). + 1. Ensure your SAML identity provider sends an attribute statement with the same name as the value of the `groups_attribute` setting. +- For GitLab.com: + 1. See [SAML SSO for GitLab.com groups](index.md). + 1. Ensure your SAML identity provider sends an attribute statement named `Groups` or `groups`. NOTE: The value for `Groups` or `groups` in the SAML response can be either the group name or the group ID. diff --git a/doc/user/group/saml_sso/img/unlink_group_saml.png b/doc/user/group/saml_sso/img/unlink_group_saml.png Binary files differdeleted file mode 100644 index 9d53a9bf407..00000000000 --- a/doc/user/group/saml_sso/img/unlink_group_saml.png +++ /dev/null diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index c05e847e2c9..80e7a5903fa 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -189,7 +189,9 @@ with the notes below for consideration. | GitLab single sign-on URL | Start URL | | Identity provider single sign-on URL | SSO URL | -You must download the certificate to get the SHA1 certificate fingerprint. +NOTE: +Google Workspace displays a SHA256 fingerprint. To retrieve the SHA1 fingerprint required by GitLab for [configuring SAML](#configure-gitlab), download the certificate and calculate +the SHA1 certificate fingerprint. The recommended attributes and claims settings are: @@ -396,9 +398,7 @@ For example, to unlink the `MyOrg` account: 1. On the top bar, in the top right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Account**. -1. In the **Social sign-in** section, select **Disconnect** next to the connected account. - -![Unlink Group SAML](img/unlink_group_saml.png) +1. In the **Service sign-in** section, select **Disconnect** next to the connected account. ## Group Sync @@ -511,7 +511,7 @@ Alternatively, the SAML response may be missing the `InResponseTo` attribute in The identity provider administrator should ensure that the login is initiated by the service provider and not the identity provider. -### Message: "Login to a GitLab account to link with your SAML identity" +### Message: "Sign in to GitLab to connect your organization's account" A user can see this message when they are trying to [manually link SAML to their existing GitLab.com account](#linking-saml-to-your-existing-gitlabcom-account). diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index cc154b96ed0..04aa99e08af 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -71,8 +71,10 @@ Follow [Azure documentation to configure the attribute mapping](https://docs.mic The following table below provides an attribute mapping known to work with GitLab. If your SAML configuration differs from [the recommended SAML settings](index.md#azure-setup-notes), -modify the corresponding `customappsso` settings accordingly. If a mapping is not listed in the -table, use the Azure defaults. For a list of required attributes, refer to the [SCIM API documentation](../../../api/scim.md). +modify the corresponding `customappsso` settings accordingly. In particular, the `externalId` must +match the [SAML NameID](index.md#nameid). +If a mapping is not listed in the table, use the Azure defaults. +For a list of required attributes, refer to the [SCIM API documentation](../../../api/scim.md). | Azure Active Directory Attribute | `customappsso` Attribute | Matching precedence | | -------------------------------- | ------------------------------ | ------------------- | @@ -169,7 +171,7 @@ If [Group SAML](index.md) has been configured and you have an existing GitLab.co We recommend users do this prior to turning on sync, because while synchronization is active, there may be provisioning errors for existing users. -New users and existing users on subsequent visits can access the group through the identify provider's dashboard or by visiting links directly. +New users and existing users on subsequent visits can access the group through the identity provider's dashboard or by visiting links directly. [In GitLab 14.0 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/325712), GitLab users created by [SAML SSO](index.md#user-access-and-management) or SCIM provisioning display with an **Enterprise** badge in the **Members** view. @@ -257,7 +259,19 @@ Changing the SAML or SCIM configuration or provider can cause the following prob | Problem | Solution | | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | SAML and SCIM identity mismatch. | First [verify that the user's SAML NameId matches the SCIM externalId](#how-do-i-verify-users-saml-nameid-matches-the-scim-externalid) and then [update or fix the mismatched SCIM externalId and SAML NameId](#update-or-fix-mismatched-scim-externalid-and-saml-nameid). | -| SCIM identity mismatch between GitLab and the Identify Provider SCIM app. | You can confirm whether you're hitting the error because of your SCIM identity mismatch between your SCIM app and GitLab.com by using [SCIM API](../../../api/scim.md#update-a-single-scim-provisioned-user) which shows up in the `id` key and compares it with the user `externalId` in the SCIM app. You can use the same [SCIM API](../../../api/scim.md#update-a-single-scim-provisioned-user) to update the SCIM `id` for the user on GitLab.com. | +| SCIM identity mismatch between GitLab and the identity provider SCIM app. | You can confirm whether you're hitting the error because of your SCIM identity mismatch between your SCIM app and GitLab.com by using [SCIM API](../../../api/scim.md#update-a-single-scim-provisioned-user) which shows up in the `id` key and compares it with the user `externalId` in the SCIM app. You can use the same [SCIM API](../../../api/scim.md#update-a-single-scim-provisioned-user) to update the SCIM `id` for the user on GitLab.com. | + +### Search Rails logs for SCIM requests + +GitLab.com administrators can search for SCIM requests in the `api_json.log` using the `pubsub-rails-inf-gprd-*` index in [Kibana](https://about.gitlab.com/handbook/support/workflows/kibana.html#using-kibana). Use the following filters based on the [SCIM API](../../../api/scim.md): + +- `json.path`: `/scim/v2/groups/<group-path>` +- `json.params.value`: `<externalId>` + +In a relevant log entry, the `json.params.value` shows the values of SCIM parameters GitLab receives. These values can be used to verify if SCIM parameters configured in an +identity provider's SCIM app are communicated to GitLab as intended. For example, we can use these values as a definitive source on why an account was provisioned with a certain +set of details. This information can help where an account was SCIM provisioned with details that appear to be incongruent with what might have been configured within an identity +provider's SCIM app. ### Azure diff --git a/doc/user/group/settings/group_access_tokens.md b/doc/user/group/settings/group_access_tokens.md index 649e7f2c264..9e8fc120731 100644 --- a/doc/user/group/settings/group_access_tokens.md +++ b/doc/user/group/settings/group_access_tokens.md @@ -78,7 +78,7 @@ or API. However, administrators can use a workaround: bot.confirm # Add the bot to the group with the required role. - group.add_user(bot, :maintainer) + group.add_member(bot, :maintainer) # Give the bot a personal access token. token = bot.personal_access_tokens.create(scopes:[:api, :write_repository], name: 'group_token') @@ -141,7 +141,7 @@ To enable or disable group access token creation for all sub-groups in a top-lev 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Settings > General**. 1. Expand **Permissions and group features**. -1. Under **Permissions**, turn on or off **Allow project and group access token creation**. +1. Under **Permissions**, turn on or off **Users can create project access tokens and group access tokens in this group**. Even when creation is disabled, you can still use and revoke existing group access tokens. diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index 5f3c859d15a..bf4e13779fd 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -73,21 +73,20 @@ To create a subgroup: To create a subgroup, you must have at least the Maintainer role on the group, depending on the group's setting. By default: -- In GitLab 12.2 or later, users with at least the Maintainer role can create subgroups. -- In GitLab 12.1 or earlier, only users with the Owner role can create subgroups. - To change who can create subgroups on a group: - As a user with the Owner role on the group: - 1. On the top bar, select **Menu > Groups** and find the group. + 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Settings > General**. 1. Expand **Permissions and group features**. - 1. Select a role from the **Allowed to create subgroups** dropdown. + 1. Select a role from **Roles allowed to create subgroups**. + 1. Select **Save changes**. - As an administrator: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Groups**. - 1. Select the group, and select **Edit**. - 1. Select a role from the **Allowed to create subgroups** dropdown. + 1. In the group's row select **Edit**. + 1. Select a role from **Allowed to create subgroups**. + 1. Select **Save changes**. For more information, view the [permissions table](../../permissions.md#group-members-permissions). diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index 72d42a8081f..3e41b7b63cc 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -7,8 +7,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Value stream analytics for groups **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196455) in GitLab 12.9 for groups. - Value stream analytics provides metrics about each stage of your software development process. A **value stream** is the entire work process that delivers value to customers. For example, @@ -20,14 +18,13 @@ Use value stream analytics to identify: - The amount of time it takes to go from an idea to production. - The velocity of a given project. - Bottlenecks in the development process. -- Detecting long-running issues or merge requests. +- Long-running issues or merge requests. - Factors that cause your software development lifecycle to slow down. Value stream analytics is also available for [projects](../../analytics/value_stream_analytics.md). ## View value stream analytics -> - Date range filter [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13216) in GitLab 12.4 > - Filtering [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13216) in GitLab 13.3 > - Horizontal stage path [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12196) in 13.0 and [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323982) in 13.12 @@ -40,7 +37,7 @@ To view value stream analytics for your group: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Analytics > Value stream**. -1. To view metrics for each stage, above the **Filter results** text box, select a stage. +1. To view metrics for a particular stage, select a stage below the **Filter results** text box. 1. Optional. Filter the results: 1. Select the **Filter results** text box. 1. Select a parameter. @@ -63,57 +60,18 @@ The table shows a list of related workflow items for the selected stage. Based o - Merge requests - Pipelines -## View metrics for each development stage +## View DORA metrics and key metrics for a group > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210315) in GitLab 13.0. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323982) in GitLab 13.12. -Value stream analytics shows the median time spent by issues or merge requests in each development stage. - -To view the median time spent in each stage by a group: - -1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Analytics > Value stream**. -1. Optional. Filter the results: - 1. Select the **Filter results** text box. - 1. Select a parameter. - 1. Select a value or enter text to refine the results. - 1. To adjust the date range: - - In the **From** field, select a start date. - - In the **To** field, select an end date. -1. To view the metrics for each stage, above the **Filter results** text box, hover over a stage. - -## View the lead time and cycle time for issues - -Value stream analytics shows the lead time and cycle time for issues in your groups: - -- Lead time: Median time from when the issue was created to when it was closed. -- Cycle time: Median time from first commit to issue closed. GitLab measures cycle time from the earliest -commit of a [linked issue's merge request](../../project/issues/crosslinking_issues.md#from-commit-messages) -to when that issue is closed. The cycle time approach underestimates the lead time because merge request creation -is always later than commit time. - -To view the lead time and cycle time for issues: - -1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Analytics > Value stream**. -1. Optional. Filter the results: - 1. Select the **Filter results** text box. - 1. Select a parameter. - 1. Select a value or enter text to refine the results. - 1. To adjust the date range: - - In the **From** field, select a start date. - - In the **To** field, select an end date. - -The **Lead Time** and **Cycle Time** metrics display below the **Filter results** text box. - -## View lead time for changes for merge requests **(ULTIMATE)** +The **Overview** dashboard in value stream analytics shows key metrics and DORA metrics of group performance. Based on the filter you select, +the dashboard automatically aggregates DORA metrics and displays the current status of the value stream. Select a DORA metric to view its chart. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340150) in GitLab 14.5. - -Lead time for changes is the median duration between when a merge request is merged and when it's deployed to production. +To view deployment metrics, you must have a +[production environment configured](../../../ci/environments/index.md#deployment-tier-of-environments). -To view the lead time for changes for merge requests in your group: +To view the DORA metrics and key metrics: 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Analytics > Value stream**. @@ -124,45 +82,46 @@ To view the lead time for changes for merge requests in your group: 1. To adjust the date range: - In the **From** field, select a start date. - In the **To** field, select an end date. +Key metrics and DORA metrics display below the **Filter results** text box. -The **Lead Time for Changes** metrics display below the **Filter results** text box. - -## View number of successful deployments **(PREMIUM)** - -> DORA API-based deployment metrics for value stream analytics for groups were [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/337256) from GitLab Ultimate to GitLab Premium in 14.3. +### Key metrics in the value stream -To view deployment metrics, you must have a -[production environment configured](../../../ci/environments/index.md#deployment-tier-of-environments). +The **Overview** dashboard shows the following key metrics that measure team performance: -Value stream analytics shows the following deployment metrics for your group: - -- Deploys: The number of successful deployments in the date range. -- Deployment Frequency: The average number of successful deployments per day in the date range. +- Lead time: Median time from when the issue was created to when it was closed. +- Cycle time: Median time from first commit to issue closed. GitLab measures cycle time from the earliest commit of a + [linked issue's merge request](../../project/issues/crosslinking_issues.md#from-commit-messages) to when that issue is closed. + The cycle time approach underestimates the lead time because merge request creation is always later than commit time. +- New issues: Number of new issues created. +- Deploys: Total number of deployments to production. -To view deployment metrics for your group: +### DORA metrics **(ULTIMATE)** -1. On the top bar, select **Menu > Groups** and find your group. -1. On the left sidebar, select **Analytics > Value stream**. -1. Optional. Filter the results: - 1. Select the **Filter results** text box. - 1. Select a parameter. - 1. Select a value or enter text to refine the results. - 1. To adjust the date range: - - In the **From** field, select a start date. - - In the **To** field, select an end date. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340150) lead time for changes DORA metric in GitLab 14.5. +> - DORA API-based deployment metrics for value stream analytics for groups were [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/337256) from GitLab Ultimate to GitLab Premium in GitLab 14.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/355304) time to restore service tile in GitLab 15.0. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/357071) change failure rate tile in GitLab 15.0. -NOTE: -The date range selector filters items by the event time. This is the time when the currently -selected stage finished for the given item. +The value stream analytics **Overview** dashboard displays the following [DORA](../../../user/analytics/index.md) metrics: -The **Deploys** and **Deployment Frequency** metrics display below the **Filter results** text box. +- Deployment Frequency. +- Lead time for changes. +- Time to restore service. +- Change failure rate. -Deployment metrics are calculated based on data from the +DORA metrics are calculated based on data from the [DORA API](../../../api/dora/metrics.md#devops-research-and-assessment-dora-key-metrics-api). NOTE: -In GitLab 13.9 and later, metrics are calculated based on when the deployment was finished. -In GitLab 13.8 and earlier, metrics are calculated based on when the deployment was created. +In GitLab 13.9 and later, deployment frequency metrics are calculated based on when the deployment was finished. +In GitLab 13.8 and earlier, deployment frequency metrics are calculated based on when the deployment was created. + +<div class="video-fallback"> + See the video: <a href="https://www.youtube.com/embed/wQU-mWvNSiI">DORA metrics and value stream analytics</a>. +</div> +<figure class="video-container"> + <iframe src="https://www.youtube.com/embed/wQU-mWvNSiI" frameborder="0" allowfullscreen="true"> </iframe> +</figure> ### How value stream analytics aggregates data @@ -186,6 +145,30 @@ longer than 10 minutes in the following cases: To view when the data was most recently updated, in the right corner next to **Edit**, hover over the **Last updated** badge. +## View metrics for each development stage + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210315) in GitLab 13.0. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323982) in GitLab 13.12. + +Value stream analytics shows the median time spent by issues or merge requests in each development stage. + +To view the median time spent in each stage by a group: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Analytics > Value stream**. +1. Optional. Filter the results: + 1. Select the **Filter results** text box. + 1. Select a parameter. + 1. Select a value or enter text to refine the results. + 1. To adjust the date range: + - In the **From** field, select a start date. + - In the **To** field, select an end date. +1. To view the metrics for each stage, above the **Filter results** text box, hover over a stage. + +NOTE: +The date range selector filters items by the event time. The event time is when the +selected stage finished for the given item. + ## How value stream analytics measures stages Value stream analytics measures each stage from its start event to its end event. @@ -207,6 +190,8 @@ Each pre-defined stages of value stream analytics is further described in the ta | Review | The median time taken to review a merge request that has a closing issue pattern, between its creation and until it's merged. | | Staging | The median time between merging a merge request that has a closing issue pattern until the very first deployment to a [production environment](#how-value-stream-analytics-identifies-the-production-environment). If there isn't a production environment, this is not tracked. | +For information about how value stream analytics calculates each stage, see the [Value stream analytics development guide](../../../development/value_stream_analytics.md). + ### Example workflow This example shows a workflow through all seven stages in one day. @@ -345,7 +330,6 @@ To delete a custom value stream: ## View number of days for a cycle to complete -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21631) in GitLab 12.6. > - Chart median line [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235455) in GitLab 13.4. > - Totals [replaced](https://gitlab.com/gitlab-org/gitlab/-/issues/262070) with averages in GitLab 13.12. @@ -367,8 +351,6 @@ The chart shows data for the last 500 workflow items. ## Tasks by type chart -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32421) in GitLab 12.10. - This chart shows a cumulative count of issues and merge requests per day. This chart uses the global page filters for displaying data based on the selected diff --git a/doc/user/infrastructure/clusters/connect/img/variables_civo.png b/doc/user/infrastructure/clusters/connect/img/variables_civo.png Binary files differindex 5a20478b13c..a668c3dd53c 100644 --- a/doc/user/infrastructure/clusters/connect/img/variables_civo.png +++ b/doc/user/infrastructure/clusters/connect/img/variables_civo.png diff --git a/doc/user/infrastructure/clusters/connect/new_civo_cluster.md b/doc/user/infrastructure/clusters/connect/new_civo_cluster.md index d8401d5a286..fad75ca6cab 100644 --- a/doc/user/infrastructure/clusters/connect/new_civo_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_civo_cluster.md @@ -64,7 +64,7 @@ Use CI/CD environment variables to configure your project. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **Variables**. -1. Set the variable `BASE64_CIVO_CREDENTIALS` to the [token](https://www.civo.com/account/security) from your Civo account. +1. Set the variable `BASE64_CIVO_TOKEN` to the [token](https://www.civo.com/account/security) from your Civo account. 1. Set the variable `TF_VAR_agent_token` to the agent token you received in the previous task. 1. Set the variable `TF_VAR_kas_address` to the agent server address in the previous task. @@ -78,8 +78,8 @@ contains other variables that you can override according to your needs: - `TF_VAR_civo_region`: Set your cluster's region. - `TF_VAR_cluster_name`: Set your cluster's name. - `TF_VAR_cluster_description`: Set a description for the cluster. To create a reference to your GitLab project on your Civo cluster detail page, set this value to `$CI_PROJECT_URL`. This value helps you determine which project was responsible for provisioning the cluster you see on the Civo dashboard. -- `TF_VAR_machine_type`: Set the machine type for the Kubernetes nodes. -- `TF_VAR_node_count`: Set the number of Kubernetes nodes. +- `TF_VAR_target_nodes_size`: Set the size of the nodes to use for the cluster +- `TF_VAR_num_target_nodes`: Set the number of Kubernetes nodes. - `TF_VAR_agent_version`: Set the version of the GitLab agent. - `TF_VAR_agent_namespace`: Set the Kubernetes namespace for the GitLab agent. diff --git a/doc/user/infrastructure/clusters/index.md b/doc/user/infrastructure/clusters/index.md index 933b310ff3f..9c8bcd9289c 100644 --- a/doc/user/infrastructure/clusters/index.md +++ b/doc/user/infrastructure/clusters/index.md @@ -13,7 +13,10 @@ To connect clusters to GitLab, use the [GitLab agent](../../clusters/agent/index WARNING: In GitLab 14.5, the certificate-based method to connect Kubernetes clusters to GitLab was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8), -as well as its related [features](#deprecated-features). +as well as its related [features](#deprecated-features). In self-managed GitLab 15.0 and later, +this feature is disabled by default. For GitLab SaaS users, this feature is available until +GitLab 15.6 for users who have at least one certificate-based cluster enabled in their namespace hierarchy. +For GitLab SaaS users that never used this feature previously, it is no longer available. The certificate-based Kubernetes integration with GitLab is deprecated. It had the following issues: @@ -40,6 +43,10 @@ for updates. You can find technical information about why we moved away from cluster certificates into the GitLab agent model on the [agent's blueprint documentation](../../../architecture/blueprints/gitlab_to_kubernetes_communication/index.md). +If you need more time to migrate to GitLab agent, you can [enable the feature flag](../../../administration/feature_flags.md) +named `certificate_based_clusters`, which was [introduced in GitLab 15.0](../../../update/deprecations.md#self-managed-certificate-based-integration-with-kubernetes). +This feature flag re-enables the certificate-based Kubernetes integration. + ## Deprecated features - [Connect an existing cluster through cluster certificates](../../project/clusters/add_existing_cluster.md) @@ -52,7 +59,6 @@ the GitLab agent model on the [agent's blueprint documentation](../../../archite - [Cluster environments](../../clusters/environments.md) - [Show Canary Ingress deployments on deploy boards](../../project/canary_deployments.md#show-canary-ingress-deployments-on-deploy-boards-deprecated) - [Deploy Boards](../../project/deploy_boards.md) -- [Pod logs](../../project/clusters/kubernetes_pod_logs.md) - [Clusters health](manage/clusters_health.md) - [Web terminals](../../../administration/integration/terminal.md) diff --git a/doc/user/infrastructure/clusters/migrate_to_gitlab_agent.md b/doc/user/infrastructure/clusters/migrate_to_gitlab_agent.md index aa07a23db18..abdb7c58d82 100644 --- a/doc/user/infrastructure/clusters/migrate_to_gitlab_agent.md +++ b/doc/user/infrastructure/clusters/migrate_to_gitlab_agent.md @@ -40,7 +40,7 @@ Some features are currently available only when using certificate-based integrat With GitLab-managed clusters, GitLab creates separate service accounts and namespaces for every branch and deploys by using these resources. -The GitLab agent uses [impersonation](../../clusters/agent/ci_cd_workflow.md#use-impersonation-to-restrict-project-and-group-access) +The GitLab agent uses [impersonation](../../clusters/agent/ci_cd_workflow.md#restrict-project-and-group-access-by-using-impersonation) strategies to deploy to your cluster with restricted account access. To do so: 1. Choose the impersonation strategy that suits your needs. diff --git a/doc/user/infrastructure/iac/terraform_state.md b/doc/user/infrastructure/iac/terraform_state.md index e8637abce91..24203e8d922 100644 --- a/doc/user/infrastructure/iac/terraform_state.md +++ b/doc/user/infrastructure/iac/terraform_state.md @@ -22,6 +22,16 @@ In GitLab, you can: - Lock and unlock states. - Remotely execute `terraform plan` and `terraform apply` commands. +WARNING: +**Disaster recovery planning** +Terraform state files are encrypted with the lockbox Ruby gem when they are at rest on disk and in object storage. +[To decrypt a state file, GitLab must be available](https://gitlab.com/gitlab-org/gitlab/-/issues/335739). +If it is offline, and you use GitLab to deploy infrastructure that GitLab requires (like virtual machines, +Kubernetes clusters, or network components), you cannot access the state file easily or decrypt it. +Additionally, if GitLab serves up Terraform modules or other dependencies that are required to bootstrap GitLab, +these will be inaccessible. To work around this issue, make other arrangements to host or back up these dependencies, +or consider using a separate GitLab instance with no shared points of failure. + ## Prerequisites For self-managed GitLab, before you can use GitLab for your Terraform state files: @@ -151,7 +161,8 @@ You can use a GitLab-managed Terraform state backend as a a [Personal Access Token](../../profile/personal_access_tokens.md) for authentication, this value is your GitLab username. If you are using GitLab CI/CD, this value is `'gitlab-ci-token'`. - **password**: The password to authenticate with the data source. If you are using a Personal Access Token for - authentication, this value is the token value. If you are using GitLab CI/CD, this value is the contents of the `${CI_JOB_TOKEN}` CI/CD variable. + authentication, this value is the token value (the token must have the **API** scope). + If you are using GitLab CI/CD, this value is the contents of the `${CI_JOB_TOKEN}` CI/CD variable. Outputs from the data source can now be referenced in your Terraform resources using `data.terraform_remote_state.example.outputs.<OUTPUT-NAME>`. diff --git a/doc/user/infrastructure/iac/troubleshooting.md b/doc/user/infrastructure/iac/troubleshooting.md index 881bcb32aed..5817337223f 100644 --- a/doc/user/infrastructure/iac/troubleshooting.md +++ b/doc/user/infrastructure/iac/troubleshooting.md @@ -112,3 +112,12 @@ job that returned the error: 1. Configure the [CI/CD environment scope](../../../ci/variables/#add-a-cicd-variable-to-a-project) for the job. 1. Set the job's [environment](../../../ci/yaml/#environment), matching the environment scope from the previous step. + +### Error refreshing state: HTTP remote state endpoint requires auth + +To resolve this, ensure that: + +- The access token you use has `api` scope. +- If you have set the `TF_HTTP_PASSWORD` CI/CD variable, make sure that you either: + - Set the same value as `TF_PASSWORD` + - Remove `TF_HTTP_PASSWORD` variable if your CI/CD job does not explicitly use it. diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index b3eadc13772..7260dbb616c 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -14,6 +14,9 @@ The Conan package registry for GitLab is under development and isn't ready for p limited functionality. This [epic](https://gitlab.com/groups/gitlab-org/-/epics/6816) details the remaining work and timelines to make it production ready. +NOTE: +The Conan registry is not FIPS compliant and is disabled when [FIPS mode](../../../development/fips_compliance.md) is enabled. + Publish Conan packages in your project's Package Registry. Then install the packages whenever you need to use them as a dependency. @@ -105,7 +108,7 @@ For more details about creating and managing Conan packages, see the #### Package without a username and a channel -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345055) in GitLab 14.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345055) in GitLab 14.6. Even though they are [recommended](https://docs.conan.io/en/latest/reference/conanfile/attributes.html#user-channel) to distinguish your package from a similarly named existing package, diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index ae64c419632..d0c771ecc41 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -208,10 +208,10 @@ should look: ```yaml build: - image: docker:19.03.12 + image: docker:20.10.16 stage: build services: - - docker:19.03.12-dind + - docker:20.10.16-dind script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY - docker build -t $CI_REGISTRY/group/project/image:latest . @@ -222,10 +222,10 @@ You can also make use of [other CI/CD variables](../../../ci/variables/index.md) ```yaml build: - image: docker:19.03.12 + image: docker:20.10.16 stage: build services: - - docker:19.03.12-dind + - docker:20.10.16-dind variables: IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG script: @@ -248,9 +248,9 @@ when needed. Changes to `main` also get tagged as `latest` and deployed using an application-specific deploy script: ```yaml -image: docker:19.03.12 +image: docker:20.10.16 services: - - docker:19.03.12-dind + - docker:20.10.16-dind stages: - build @@ -323,9 +323,9 @@ Below is an example of what your `.gitlab-ci.yml` should look like: ```yaml build: - image: $CI_REGISTRY/group/project/docker:19.03.12 + image: $CI_REGISTRY/group/project/docker:20.10.16 services: - - name: $CI_REGISTRY/group/project/docker:19.03.12-dind + - name: $CI_REGISTRY/group/project/docker:20.10.16-dind alias: docker stage: build script: @@ -333,7 +333,7 @@ build: - docker run my-docker-image /script/to/run/tests ``` -If you forget to set the service alias, the `docker:19.03.12` image is unable to find the +If you forget to set the service alias, the `docker:20.10.16` image is unable to find the `dind` service, and an error like the following is thrown: ```plaintext @@ -353,7 +353,7 @@ Below is an example of what your `.gitlab-ci.yml` should look like: ```yaml build: - image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker:19.03.12 + image: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker:20.10.16 services: - name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker:18.09.7-dind alias: docker @@ -363,7 +363,7 @@ build: - docker run my-docker-image /script/to/run/tests ``` -If you forget to set the service alias, the `docker:19.03.12` image is unable to find the +If you forget to set the service alias, the `docker:20.10.16` image is unable to find the `dind` service, and an error like the following is thrown: ```plaintext @@ -438,10 +438,10 @@ stages: - clean build_image: - image: docker:19.03.12 + image: docker:20.10.16 stage: build services: - - docker:19.03.12-dind + - docker:20.10.16-dind variables: IMAGE_TAG: $CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG script: @@ -454,10 +454,10 @@ build_image: - main delete_image: - image: docker:19.03.12 + image: docker:20.10.16 stage: clean services: - - docker:19.03.12-dind + - docker:20.10.16-dind variables: IMAGE_TAG: $CI_PROJECT_PATH:$CI_COMMIT_REF_SLUG REG_SHA256: ade837fc5224acd8c34732bf54a94f579b47851cc6a7fd5899a98386b782e228 @@ -564,7 +564,7 @@ project or branch name. Special characters can include: - Trailing hyphen/dash To get around this, you can [change the group path](../../group/index.md#change-a-groups-path), -[change the project path](../../project/settings/index.md#renaming-a-repository) or change the branch +[change the project path](../../project/settings/index.md#rename-a-repository) or change the branch name. You may also get a `404 Not Found` or `Unknown Manifest` message if you are using diff --git a/doc/user/packages/debian_repository/index.md b/doc/user/packages/debian_repository/index.md index a8f0672e376..1d846a60281 100644 --- a/doc/user/packages/debian_repository/index.md +++ b/doc/user/packages/debian_repository/index.md @@ -15,6 +15,9 @@ The Debian package registry for GitLab is under development and isn't ready for limited functionality. This [epic](https://gitlab.com/groups/gitlab-org/-/epics/6057) details the remaining work and timelines to make it production ready. +NOTE: +The Debian registry is not FIPS compliant and is disabled when [FIPS mode](../../../development/fips_compliance.md) is enabled. + Publish Debian packages in your project's Package Registry. Then install the packages whenever you need to use them as a dependency. diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index af54d928bec..4770057e4ea 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -152,17 +152,17 @@ FROM gitlab.example.com:443/my-group/dependency_proxy/containers/alpine:latest ```yaml # .gitlab-ci.yml -image: docker:19.03.12 +image: docker:20.10.16 variables: DOCKER_HOST: tcp://docker:2375 DOCKER_TLS_CERTDIR: "" services: - - docker:19.03.12-dind + - docker:20.10.16-dind build: - image: docker:19.03.12 + image: docker:20.10.16 before_script: - docker login -u $CI_DEPENDENCY_PROXY_USER -p $CI_DEPENDENCY_PROXY_PASSWORD $CI_DEPENDENCY_PROXY_SERVER script: @@ -301,7 +301,7 @@ hub_docker_quota_check: ### Dependency Proxy Connection Failure -If a service alias is not set the `docker:19.03.12` image is unable to find the +If a service alias is not set the `docker:20.10.16` image is unable to find the `dind` service, and an error like the following is thrown: ```plaintext diff --git a/doc/user/packages/generic_packages/index.md b/doc/user/packages/generic_packages/index.md index 22b792e443f..8d5fc73ad4e 100644 --- a/doc/user/packages/generic_packages/index.md +++ b/doc/user/packages/generic_packages/index.md @@ -209,3 +209,25 @@ upload: The [Write CI-CD Variables in Pipeline](https://gitlab.com/guided-explorations/cfg-data/write-ci-cd-variables-in-pipeline) project contains a working example you can use to create, upload, and download generic packages in GitLab CI/CD. It also demonstrates how to manage a semantic version for the generic package: storing it in a CI/CD variable, retrieving it, incrementing it, and writing it back to the CI/CD variable when tests for the download work correctly. + +## Troubleshooting + +### Internal Server error on large file uploads to S3 + +S3-compatible object storage [limits the size of a single PUT request to 5GB](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html). If the `aws_signature_version` is set to `2` in the [object storage connection settings](../../../administration/object_storage.md), attempting to publish a package file larger than the 5GB limit can result in a `HTTP 500: Internal Server Error` response. + +If you are receiving `HTTP 500: Internal Server Error` responses when publishing large files to S3, set the `aws_signature_version` to `4`: + +```ruby +# Consolidated Object Storage settings +gitlab_rails['object_store']['connection'] = { + # Other connection settings + `aws_signature_version` => '4' +} +# OR +# Storage-specific form settings +gitlab_rails['packages_object_store_connection'] = { + # Other connection settings + `aws_signature_version` => '4' +} +``` diff --git a/doc/user/packages/helm_repository/index.md b/doc/user/packages/helm_repository/index.md index 88ea5afad3c..07e853fa18c 100644 --- a/doc/user/packages/helm_repository/index.md +++ b/doc/user/packages/helm_repository/index.md @@ -69,6 +69,11 @@ Once built, a chart can be uploaded to the desired channel with `curl` or `helm - `<project_id>`: the project ID (like `42`). - `<channel>`: the name of the channel (like `stable`). +### Release channels + +You can publish Helm charts to channels in GitLab. Channels are a method you can use to differentiate Helm chart repositories. +For example, you can use `stable` and `devel` as channels to allow users to add the `stable` repo while `devel` charts are isolated. + ## Use CI/CD to publish a Helm package To publish a Helm package automated through [GitLab CI/CD](../../../ci/index.md), you can use diff --git a/doc/user/packages/infrastructure_registry/index.md b/doc/user/packages/infrastructure_registry/index.md index 551289a575a..e6a179c9d12 100644 --- a/doc/user/packages/infrastructure_registry/index.md +++ b/doc/user/packages/infrastructure_registry/index.md @@ -85,3 +85,19 @@ You can also remove the Infrastructure Registry for a specific project: 1. Select **Save changes**. To enable it back, follow the same steps above and toggle it on (in blue). + +## How module resolution works + +When you upload a new module, GitLab generates a path for the module, for example, `https://gitlab.example.com/parent-group/my-infra-package`. + +- This path conforms with [the Terraform spec](https://www.terraform.io/internals/module-registry-protocol). +- The name of the path must be unique within the namespace. + +For projects in subgroups, GitLab checks that the module name does not already exist anywhere in the namespace, including all subgroups and the parent group. + +For example, if: + +- The project is `gitlab.example.com/parent-group/sub-group/my-project`. +- The infrastructure package is `my-infra-package`. + +The project name must be unique in all projects in all groups under `parent-group`. diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index bdcbea68568..7ea3c1aa0c8 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -145,6 +145,9 @@ If you encounter an error with [Yarn](https://classic.yarnpkg.com/en/), view #### Instance-level npm endpoint +NOTE: +Note: Using `CI_JOB_TOKEN` to install npm packages with dependencies in another project will give you 404 errors. You can use a [personal access token](../../profile/personal_access_tokens.md) as a workaround. [GitLab-#352962](https://gitlab.com/gitlab-org/gitlab/-/issues/352962) proposes a fix to this bug. + To use the [instance-level](#use-the-gitlab-endpoint-for-npm-packages) npm endpoint, set your npm configuration: ```shell @@ -232,6 +235,12 @@ When you use the [instance-level endpoint](#use-the-gitlab-endpoint-for-npm-pack example, a package `@MyScope/package-name` in GitLab becomes `@myscope/package-name` for npm. - The `package-name` can be whatever you want. +NOTE: +The value used for the `@scope` is the root of the project that will end up hosting the packages and not the root +of the project with the source code of the package itself. For example, assume your package source code is located +at `source-code-group/package-code` and deployed to a package registry inside `registries-group/registry-project`. +In this case, the `@scope` needs to be `@registries-group` and not `@source-code-group`. + For example, if your project is `https://gitlab.example.com/my-org/engineering-group/team-amazing/analytics`, the root namespace is `my-org`. When you publish a package, it must have `my-org` as the scope. diff --git a/doc/user/packages/package_registry/reduce_package_registry_storage.md b/doc/user/packages/package_registry/reduce_package_registry_storage.md index ed4ef1665bc..4a03bd9e8a0 100644 --- a/doc/user/packages/package_registry/reduce_package_registry_storage.md +++ b/doc/user/packages/package_registry/reduce_package_registry_storage.md @@ -50,3 +50,39 @@ To delete package files in the UI, from your group or project: 1. Expand the ellipsis and select **Delete file**. The package files are permanently deleted. + +## Cleanup policy + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346153) in GitLab 15.2. + +Depending on the number of packages to remove, the process of manually deleting the packages can take a long time to finish. +A cleanup policy defines a set of rules that, applied to a project, defines which package files you can automatically delete. + +### Enable the cleanup policy + +By default, the packages cleanup policy is disabled. To enable it: + +1. Go to your project **Settings > Packages & Registries**. +1. Expand **Manage storage used by package assets**. +1. Set the rules appropriately. + +NOTE: +To access these project settings, you must be at least a maintainer on the related project. + +### Available rules + +- `Number of duplicated assets to keep`. The number of duplicated assets to keep. Some package formats allow you + to upload more than one copy of an asset. You can limit the number of duplicated assets to keep and automatically + delete the oldest files once the limit is reached. + +### Set cleanup limits to conserve resources + +A background process executes the package-cleanup policies. This process can take a long time to finish and consumes +server resources while it is running. + +You can use the following setting to limit the number of cleanup workers: + +- `package_registry_cleanup_policies_worker_capacity`: the maximum number of cleanup workers running concurrently. + This number must be greater than or equal to `0`. + We recommend starting with a low number and increasing it after monitoring the resources used by the background workers. + To remove all workers and not execute the cleanup policies, set this to `0`. The default value is `2`. diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index eee6d55a3ce..b8996dc2963 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -99,45 +99,54 @@ Hello from MyPyPiPackage After you create a project, you can create a package. 1. In your terminal, go to the `MyPyPiPackage` directory. -1. Create a `setup.py` file: +1. Create a `pyproject.toml` file: ```shell - touch setup.py + touch pyproject.toml ``` This file contains all the information about the package. For more information - about this file, see [creating setup.py](https://packaging.python.org/tutorials/packaging-projects/#creating-setup-py). + about this file, see [creating `pyproject.toml`](https://packaging.python.org/en/latest/tutorials/packaging-projects/#creating-pyproject-toml). Because GitLab identifies packages based on [Python normalized names (PEP-503)](https://www.python.org/dev/peps/pep-0503/#normalized-names), ensure your package name meets these requirements. See the [installation section](#authenticate-with-a-ci-job-token) for details. -1. Open the `setup.py` file, and then add basic information: - - ```python - import setuptools - - setuptools.setup( - name="mypypipackage", - version="0.0.1", - author="Example Author", - author_email="author@example.com", - description="A small example package", - packages=setuptools.find_packages(), - classifiers=[ - "Programming Language :: Python :: 3", - "License :: OSI Approved :: MIT License", - "Operating System :: OS Independent", - ], - python_requires='>=3.6', - ) +1. Open the `pyproject.toml` file, and then add basic information: + + ```toml + [build-system] + requires = ["setuptools>=61.0"] + build-backend = "setuptools.build_meta" + + [project] + name = "mypypipackage" + version = "0.0.1" + authors = [ + { name="Example Author", email="author@example.com" }, + ] + description = "A small example package" + requires-python = ">=3.7" + classifiers = [ + "Programming Language :: Python :: 3", + "Operating System :: OS Independent", + ] + + [tool.setuptools.packages] + find = {} ``` 1. Save the file. -1. Execute the setup: +1. Install the package build library: + + ```shell + pip install build + ``` + +1. Build the package: ```shell - python3 setup.py sdist bdist_wheel + python -m build ``` The output should be visible in a newly-created `dist` folder: @@ -218,8 +227,8 @@ image: python:latest run: script: - - pip install twine - - python setup.py sdist bdist_wheel + - pip install build twine + - python -m build - TWINE_PASSWORD=${CI_JOB_TOKEN} TWINE_USERNAME=gitlab-ci-token python -m twine upload --repository-url ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/pypi dist/* ``` diff --git a/doc/user/packages/terraform_module_registry/index.md b/doc/user/packages/terraform_module_registry/index.md index 42c85ae9d41..2668b8b35ac 100644 --- a/doc/user/packages/terraform_module_registry/index.md +++ b/doc/user/packages/terraform_module_registry/index.md @@ -108,7 +108,7 @@ Where `<namespace>` is the [namespace](../../../user/group/index.md#namespaces) To work with Terraform modules in [GitLab CI/CD](../../../ci/index.md), you can use `CI_JOB_TOKEN` in place of the personal access token in your commands. -For example: +For example, this job uploads a new module for the `local` [system provider](https://registry.terraform.io/browse/providers) and uses the module version from the Git commit tag: ```yaml stages: @@ -121,15 +121,18 @@ upload: TERRAFORM_MODULE_DIR: ${CI_PROJECT_DIR} # The path to your Terraform module TERRAFORM_MODULE_NAME: ${CI_PROJECT_NAME} # The name of your Terraform module TERRAFORM_MODULE_SYSTEM: local # The system or provider your Terraform module targets (ex. local, aws, google) - TERRAFORM_MODULE_VERSION: ${CI_COMMIT_TAG} # The version of your Terraform module to be published to your project's registry + TERRAFORM_MODULE_VERSION: ${CI_COMMIT_TAG} # Tag commits with SemVer for the version of your Terraform module to be published script: - - tar -cvzf ${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz -C ${TERRAFORM_MODULE_DIR} --exclude=./.git . - - 'curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --upload-file ${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/terraform/modules/${TERRAFORM_MODULE_NAME}/${TERRAFORM_MODULE_SYSTEM}/${TERRAFORM_MODULE_VERSION}/file' + - TERRAFORM_MODULE_NAME=$(echo "${TERRAFORM_MODULE_NAME}" | tr " _" -) # module-name must not have spaces or underscores, so translate them to hyphens + - tar -vczf ${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz -C ${TERRAFORM_MODULE_DIR} --exclude=./.git . + - 'curl --location --header "JOB-TOKEN: ${CI_JOB_TOKEN}" + --upload-file ${TERRAFORM_MODULE_NAME}-${TERRAFORM_MODULE_SYSTEM}-${TERRAFORM_MODULE_VERSION}.tgz + ${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/terraform/modules/${TERRAFORM_MODULE_NAME}/${TERRAFORM_MODULE_SYSTEM}/${TERRAFORM_MODULE_VERSION}/file' rules: - if: $CI_COMMIT_TAG ``` -To trigger this upload job, add a Git tag to your commit. The `rules:if: $CI_COMMIT_TAG` defines this so that not every commit to your repo triggers the upload. +To trigger this upload job, add a Git tag to your commit. Ensure the tag follows the [Semantic Versioning Specification](https://semver.org/) that Terraform requires. The `rules:if: $CI_COMMIT_TAG` ensures that only tagged commits to your repo trigger the module upload job. For other ways to control jobs in your CI/CD pipeline, refer to the [`.gitlab-ci.yml`](../../../ci/yaml/index.md) keyword reference. ## Example projects diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 801c107e371..b01bfbef3aa 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -63,13 +63,12 @@ The following table lists project permissions available for each role: | [Analytics](analytics/index.md):<br>View [CI/CD analytics](analytics/ci_cd_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | [Analytics](analytics/index.md):<br>View [code review analytics](analytics/code_review_analytics.md) | | ✓ | ✓ | ✓ | ✓ | | [Analytics](analytics/index.md):<br>View [repository analytics](analytics/repository_analytics.md) | | ✓ | ✓ | ✓ | ✓ | -| [Application security](application_security/index.md):<br>View licenses in [dependency list](application_security/dependency_list/index.md) | ✓ (*1*) | ✓ | ✓ | ✓ | ✓ | +| [Application security](application_security/index.md):<br>View licenses in [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create and run [on-demand DAST scans](application_security/dast/index.md#on-demand-scans) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Manage [security policy](application_security/policies/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>View [dependency list](application_security/dependency_list/index.md) | | | ✓ | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create a [CVE ID Request](application_security/cve_id_request.md) | | | | ✓ | ✓ | | [Application security](application_security/index.md):<br>Create or assign [security policy project](application_security/policies/index.md) | | | | | ✓ | -| [Clusters](infrastructure/clusters/index.md):<br>View [pod logs](project/clusters/kubernetes_pod_logs.md) | | | ✓ | ✓ | ✓ | | [Clusters](infrastructure/clusters/index.md):<br>View clusters | | | ✓ | ✓ | ✓ | | [Clusters](infrastructure/clusters/index.md):<br>Manage clusters | | | | ✓ | ✓ | | [Container Registry](packages/container_registry/index.md):<br>Create, edit, delete [cleanup policies](packages/container_registry/index.md#delete-images-by-using-a-cleanup-policy) | | | | ✓ | ✓ | @@ -265,7 +264,7 @@ More details about the permissions for some project-level features follow. | View pipeline details page | ✓ (*1*) | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | | View pipelines page | ✓ (*1*) | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | | View pipelines tab in MR | ✓ (*3*) | ✓ (*3*) | ✓ | ✓ | ✓ | ✓ | -| [View vulnerabilities in a pipeline](application_security/security_dashboard/index.md#view-vulnerabilities-in-a-pipeline) | | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | +| [View vulnerabilities in a pipeline](application_security/vulnerability_report/pipeline.md#view-vulnerabilities-in-a-pipeline) | | ✓ (*2*) | ✓ | ✓ | ✓ | ✓ | | View and download project-level [Secure Files](../api/secure_files.md) | | | | ✓ | ✓ | ✓ | | Cancel and retry jobs | | | | ✓ | ✓ | ✓ | | Create new [environments](../ci/environments/index.md) | | | | ✓ | ✓ | ✓ | @@ -432,6 +431,7 @@ The following table lists group permissions available for each role: | View group [Usage Quotas](usage_quotas.md) page | | | | | ✓ (4) | | Manage group runners | | | | | ✓ | | [Migrate groups](group/import/index.md) | | | | | ✓ | +| Manage [subscriptions, and purchase CI/CD minutes and storage](../subscriptions/gitlab_com/index.md) | | | | | ✓ | <!-- markdownlint-disable MD029 --> diff --git a/doc/user/profile/account/create_accounts.md b/doc/user/profile/account/create_accounts.md index 7e1074aa50f..694ed02a694 100644 --- a/doc/user/profile/account/create_accounts.md +++ b/doc/user/profile/account/create_accounts.md @@ -9,32 +9,35 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can create users: -- Manually through the sign in page or Administrator Area. -- Automatically through user authentication integrations. +- [Manually through the sign-in page](#create-users-on-sign-in-page). +- [Manually in the Admin Area](#create-users-in-admin-area). +- [Manually using the API](../../../api/users.md). +- [Automatically through user authentication integrations](#create-users-through-authentication-integrations). -## Create users on sign in page +## Create users on sign-in page -If you have [sign-up enabled](../../admin_area/settings/sign_up_restrictions.md), users can create -their own accounts by either: +Prerequisites: -- Selecting the **Register now** link on the sign-in page. -- Navigating to `https://gitlab.example.com/users/sign_up`. +- [Sign-up enabled](../../admin_area/settings/sign_up_restrictions.md) + +Users can create their own accounts by either: -![Register Tab](img/register_v13_6.png) +- Selecting the **Register now** link on the sign-in page. +- Navigating to your GitLab instance's sign-up link. For example: `https://gitlab.example.com/users/sign_up`. ## Create users in Admin Area -As an Administrator user, you can manually create users: +Prerequisites: + +- You must have administrator access for the instance. + +To create a user manually: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Users** (`/admin/users`). 1. Select **New user**. - -You can also [create users through the API](../../../api/users.md) as an administrator. - -![Administrator User Button](img/admin_user_button.png) - -![Administrator User Form](img/admin_user_form.png) +1. Complete the fields. +1. Select **Create user**. ## Create users through authentication integrations diff --git a/doc/user/profile/account/img/admin_user_button.png b/doc/user/profile/account/img/admin_user_button.png Binary files differdeleted file mode 100644 index 506e16bb8ca..00000000000 --- a/doc/user/profile/account/img/admin_user_button.png +++ /dev/null diff --git a/doc/user/profile/account/img/admin_user_form.png b/doc/user/profile/account/img/admin_user_form.png Binary files differdeleted file mode 100644 index aebc31ee3ff..00000000000 --- a/doc/user/profile/account/img/admin_user_form.png +++ /dev/null diff --git a/doc/user/profile/account/img/register_v13_6.png b/doc/user/profile/account/img/register_v13_6.png Binary files differdeleted file mode 100644 index ce4adc0f55b..00000000000 --- a/doc/user/profile/account/img/register_v13_6.png +++ /dev/null diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index 2dbeaae2267..4563cfe5648 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -59,11 +59,12 @@ To enable 2FA with a one-time password: 1. Select **Enable Two-factor Authentication**. 1. **On your device (usually your phone):** 1. Install a compatible application. For example: + - [Aegis](https://getaegis.app/) + - [Raivo OTP](https://apps.apple.com/us/app/raivo-otp/id1459042137#platform=iphone) - [Authy](https://authy.com/) - [Duo Mobile](https://duo.com/product/multi-factor-authentication-mfa/duo-mobile-app) - [LastPass Authenticator](https://lastpass.com/auth/) - [Authenticator](https://mattrubin.me/authenticator/) - - [andOTP](https://github.com/andOTP/andOTP) - [Google Authenticator](https://support.google.com/accounts/answer/1066447?hl=en) - [Microsoft Authenticator](https://www.microsoft.com/en-us/security/mobile-authenticator-app) - [SailOTP](https://openrepos.net/content/seiichiro0185/sailotp) diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index 07f21da3099..bf696310158 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -130,7 +130,7 @@ GitLab displays the contents of your README below your contribution graph. ### From an existing project To add the README from an existing project to your profile, -[update the path](../project/settings/index.md#renaming-a-repository) of the project +[update the path](../project/settings/index.md#rename-a-repository) of the project to match your username. ## Add external accounts to your user profile page diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index af84b746280..427c412219a 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -145,7 +145,7 @@ To create a personal access token programmatically: ``` This code can be shortened into a single-line shell command by using the -[Rails runner](../../administration/troubleshooting/debug.md#using-the-rails-runner): +[Rails runner](../../administration/operations/rails_console.md#using-the-rails-runner): ```shell sudo gitlab-rails runner "token = User.find_by_username('automation-bot').personal_access_tokens.create(scopes: [:read_user, :read_repository], name: 'Automation token'); token.set_token('token-string-here123'); token.save!" @@ -177,7 +177,7 @@ To revoke a token programmatically: ``` This code can be shortened into a single-line shell command using the -[Rails runner](../../administration/troubleshooting/debug.md#using-the-rails-runner): +[Rails runner](../../administration/operations/rails_console.md#using-the-rails-runner): ```shell sudo gitlab-rails runner "PersonalAccessToken.find_by_token('token-string-here123').revoke!" diff --git a/doc/user/project/clusters/img/kubernetes_pod_logs_v12_10.png b/doc/user/project/clusters/img/kubernetes_pod_logs_v12_10.png Binary files differdeleted file mode 100644 index abac22e3f1f..00000000000 --- a/doc/user/project/clusters/img/kubernetes_pod_logs_v12_10.png +++ /dev/null diff --git a/doc/user/project/clusters/img/pod_logs_deploy_board.png b/doc/user/project/clusters/img/pod_logs_deploy_board.png Binary files differdeleted file mode 100644 index 7f83382968b..00000000000 --- a/doc/user/project/clusters/img/pod_logs_deploy_board.png +++ /dev/null diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md index 58006c29057..bd87ab1024d 100644 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -2,120 +2,11 @@ stage: Monitor group: Respond info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +remove_date: '2022-18-10' +redirect_to: '../../clusters/agent/index.md' --- -# Kubernetes Logs (DEPRECATED) **(FREE SELF)** +# Kubernetes Logs (removed) **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4752) in GitLab 11.0. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26383) from GitLab Ultimate to GitLab Free 12.9. -> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/360182) behind a [feature flag](../../../administration/feature_flags.md) named `monitor_logging` in GitLab 15.0. Disabled by default. -> - [Disabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/353410) in GitLab 15.0. - -WARNING: -This feature is in its end-of-life process. -This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -It will be [removed completely](https://gitlab.com/gitlab-org/gitlab/-/issues/346485) in GitLab 15.2. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `monitor_logging` and the one named `certificate_based_clusters`. -On GitLab.com, this feature is not available. -This feature is not recommended for production use. - -GitLab makes it easy to view the logs of running pods in -[connected Kubernetes clusters](index.md). By displaying the logs directly in GitLab -in the **Log Explorer**, developers can avoid managing console tools or jumping -to a different interface. The **Log Explorer** interface provides a set of filters -above the log file data, depending on your configuration: - -![Pod logs](img/kubernetes_pod_logs_v12_10.png) - -- **Namespace** - Select the environment to display. Users with Maintainer or - greater [permissions](../../permissions.md) can also see pods in the - `gitlab-managed-apps` namespace. -- **Search** - Only available if the [Elastic Stack integration](../../clusters/integrations.md#elastic-stack-cluster-integration) is enabled. -- **Select time range** - Select the range of time to display. - Only available if the [Elastic Stack integration](../../clusters/integrations.md#elastic-stack-cluster-integration) is enabled. -- **Scroll to bottom** **{scroll_down}** - Scroll to the end of the displayed logs. -- **Refresh** **{retry}** - Reload the displayed logs. - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> -To learn more about the Log Explorer, see [APM - Log Explorer](https://www.youtube.com/watch?v=hWclZHA7Dgw). - -[Learn more about Kubernetes + GitLab](https://about.gitlab.com/solutions/kubernetes/). -Everything you need to build, test, deploy, and run your application at scale. - -## Requirements - -[Deploying to a Kubernetes environment](../deploy_boards.md#enabling-deploy-boards) -is required to use Logs. - -## Accessing the log explorer - -To access the **Log explorer**, select the **More actions** **{ellipsis_v}** menu on -a [metrics dashboard](../../../operations/metrics/index.md) and select **View logs**, or: - -1. Sign in as a user with the _View pod logs_ - [permissions](../../permissions.md#project-members-permissions) in the project. -1. To navigate to the **Log Explorer** from the sidebar menu, go to **Monitor > Logs** - ([Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22011) in GitLab 12.5.). -1. To navigate to the **Log Explorer** from a specific pod on a [deploy board](../deploy_boards.md): - - 1. Go to **Deployments > Environments** and find the environment - which contains the desired pod, like `production`. - 1. On the **Environments** page, you should see the status of the environment's - pods with [deploy boards](../deploy_boards.md). - 1. When mousing over the list of pods, GitLab displays a tooltip with the exact pod name - and status. - ![deploy boards pod list](img/pod_logs_deploy_board.png) - 1. Select the desired pod to display the **Log Explorer**. - -### Logs view - -The **Log Explorer** lets you filter the logs by: - -- Pods. -- [From GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/5769), environments. -- [From GitLab 12.7](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21656), - [full text search](#full-text-search). -- [From GitLab 12.8](https://gitlab.com/gitlab-org/gitlab/-/issues/197879), dates. -- [From GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/208790), managed apps. - -Loading more than 500 log lines is possible from -[GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/198050) onward. - -Support for pods with multiple containers is coming -[in a future release](https://gitlab.com/gitlab-org/gitlab/-/issues/13404). - -Support for historical data is coming -[in a future release](https://gitlab.com/gitlab-org/gitlab/-/issues/196191). - -### Filter by date - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/197879) in GitLab 12.8. - -When you enable [Elastic Stack](../../clusters/integrations.md#elastic-stack-cluster-integration) -on your cluster, you can filter logs displayed in the **Log Explorer** by date. - -Select **Show last** in the **Log Explorer** to see the available options. - -### Full text search - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21656) in GitLab 12.7. - -When you enable [Elastic Stack](../../clusters/integrations.md#elastic-stack-cluster-integration) on your cluster, -you can search the content of your logs through a search bar. The search is passed -to Elasticsearch using the -[simple_query_string](https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html) -Elasticsearch function, which supports the following operators: - -| Operator | Description | -|----------------------------|-------------------------------------------------------------| -| `\|` | An `OR` operation. | -| `-` | Negates a single token. | -| `+` | An `AND` operation. | -| `"` | Wraps a number of tokens to signify a phrase for searching. | -| `*` (at the end of a term) | A prefix query. | -| `(` and `)` | Precedence. | -| `~N` (after a word) | Edit distance (fuzziness). | -| `~N` (after a phrase) | Slop amount. | +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5 +and [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/360193) in GitLab 15.2. diff --git a/doc/user/project/code_owners.md b/doc/user/project/code_owners.md index 197a995952a..adea5dad7b8 100644 --- a/doc/user/project/code_owners.md +++ b/doc/user/project/code_owners.md @@ -85,6 +85,12 @@ Inviting **Subgroup Y** to a parent group of **Project A** [is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/288851). To set **Subgroup Y** as Code Owners, add this group directly to the project itself. +NOTE: +For approval to be required, groups as Code Owners must have a direct membership +(not inherited membership) in the project. Approval can only be optional for groups +that inherit membership. Members in the Code Owners group also must be direct members, +and not inherit membership from any parent groups. + ### Add a group as a Code Owner To set a group as a Code Owner: diff --git a/doc/user/project/deploy_keys/index.md b/doc/user/project/deploy_keys/index.md index 8f1da4b278a..c64afd95d8d 100644 --- a/doc/user/project/deploy_keys/index.md +++ b/doc/user/project/deploy_keys/index.md @@ -82,7 +82,7 @@ Prerequisites: A project deploy key is enabled when it is created. You can modify only a project deploy key's name and permissions. -## Create a public deploy key +## Create a public deploy key **(FREE SELF)** Prerequisites: diff --git a/doc/user/project/highlighting.md b/doc/user/project/highlighting.md index 37ec7c8e8d3..1d62cd00b31 100644 --- a/doc/user/project/highlighting.md +++ b/doc/user/project/highlighting.md @@ -7,7 +7,7 @@ type: reference # Syntax Highlighting **(FREE)** -GitLab provides syntax highlighting on all files through the +GitLab provides syntax highlighting on all files through [Highlight.js](https://github.com/highlightjs/highlight.js/) and the [Rouge](https://rubygems.org/gems/rouge) Ruby gem. It attempts to guess what language to use based on the file extension, which most of the time is sufficient. diff --git a/doc/user/project/img/labels_drag_priority_v12_1.gif b/doc/user/project/img/labels_drag_priority_v12_1.gif Binary files differdeleted file mode 100644 index a568490da5f..00000000000 --- a/doc/user/project/img/labels_drag_priority_v12_1.gif +++ /dev/null diff --git a/doc/user/project/img/time_tracking_report_v15_1.png b/doc/user/project/img/time_tracking_report_v15_1.png Binary files differindex a9ddefebb2f..4eeccf8a684 100644 --- a/doc/user/project/img/time_tracking_report_v15_1.png +++ b/doc/user/project/img/time_tracking_report_v15_1.png diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index b2425686024..dff5a602e8a 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -58,7 +58,7 @@ For user contributions to be mapped, each user must complete the following befor If they don't match, modify the public name in the Atlassian account settings to match the username in the Bitbucket account settings. -1. Connect your Bitbucket account in [GitLab profile social sign-in](https://gitlab.com/-/profile/account). +1. Connect your Bitbucket account in [GitLab profile service sign-in](https://gitlab.com/-/profile/account). 1. [Set your public email](../../profile/index.md#set-your-public-email). @@ -97,16 +97,16 @@ If you've accidentally started the import process with the wrong account, follow the username in the Bitbucket account settings must match the public name in the Atlassian account settings. If these names match but user mapping still fails, the user may have modified their Bitbucket username after connecting their Bitbucket account in the -[GitLab profile social sign-in](https://gitlab.com/-/profile/account). +[GitLab profile service sign-in](https://gitlab.com/-/profile/account). To fix this, the user must verify that their Bitbucket external UID in the GitLab database matches their current Bitbucket public name, and reconnect if there's a mismatch: -1. [Use the API to get the currently authenticated user](../../../api/users.md#list-current-user-for-normal-users). +1. [Use the API to get the currently authenticated user](../../../api/users.md#for-normal-users-1). 1. In the API's response, the `identities` attribute contains the Bitbucket account that exists in the GitLab database. If the `extern_uid` doesn't match the current Bitbucket public name, the - user should reconnect their Bitbucket account in the [GitLab profile social sign-in](https://gitlab.com/-/profile/account). + user should reconnect their Bitbucket account in the [GitLab profile service sign-in](https://gitlab.com/-/profile/account). 1. Following reconnection, the user should use the API again to verify that their `extern_uid` in the GitLab database now matches their current Bitbucket public name. diff --git a/doc/user/project/index.md b/doc/user/project/index.md index 60a4ca5c0ea..e4ae0c4b29b 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -150,7 +150,7 @@ There are numerous [APIs](../../api/index.md) to use with your projects: - [Traffic](../../api/project_statistics.md) - [Variables](../../api/project_level_variables.md) - [Aliases](../../api/project_aliases.md) -- [DORA4 Analytics](../../api/dora4_project_analytics.md) +- [DORA4 Analytics](../../api/dora/metrics.md) ## DORA4 analytics overview @@ -158,4 +158,4 @@ Project details include the following analytics: - Deployment Frequency -For more information, see [DORA4 Project Analytics API](../../api/dora4_project_analytics.md). +For more information, see [DORA4 Project Analytics API](../../api/dora/metrics.md). diff --git a/doc/user/project/integrations/bamboo.md b/doc/user/project/integrations/bamboo.md index 22e6d45dd96..75f099268cb 100644 --- a/doc/user/project/integrations/bamboo.md +++ b/doc/user/project/integrations/bamboo.md @@ -9,11 +9,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can automatically trigger builds in Atlassian Bamboo when you push changes to your project in GitLab. -When this integration is configured, merge requests also display the following information: - -- A CI/CD status that shows if the build is pending, failed, or has completed successfully. -- A link to the Bamboo build page for more information. - Bamboo doesn't provide the same features as a traditional build system when accepting webhooks and commit data. You must configure a Bamboo build plan before you configure the integration in GitLab. @@ -66,6 +61,65 @@ for example `PROJ-PLAN`. The build key is included in the browser URL when you view a plan in Bamboo. For example, `https://bamboo.example.com/browse/PROJ-PLAN`. +## Update Bamboo build status in GitLab + +You can use a script that uses the [commit status API](../../../api/commits.md#post-the-build-status-to-a-commit) +and Bamboo build variables to: + +- Update the commit with the build status. +- Add the Bamboo build plan URL as the commit's `target_url`. + +For example: + +1. Create an [access token](../../../api/index.md#personalprojectgroup-access-tokens) in GitLab with `:api` permissions. +1. Save the token as a `$GITLAB_TOKEN` variable in Bamboo. +1. Add the following script as a final task to the Bamboo plan's jobs: + + ```shell + #!/bin/bash + + # Script to update CI status on GitLab. + # Add this script as final inline script task in a Bamboo job. + # + # General documentation: https://docs.gitlab.com/ee/user/project/integrations/bamboo.html + # Fix inspired from https://gitlab.com/gitlab-org/gitlab/-/issues/34744 + + # Stop at first error + set -e + + # Access token. Set this as a CI variable in Bamboo. + #GITLAB_TOKEN= + + # Status + cistatus="failed" + if [ "${bamboo_buildFailed}" = "false" ]; then + cistatus="success" + fi + + repo_url="${bamboo_planRepository_repositoryUrl}" + + # Check if we use SSH or HTTPS + protocol=${repo_url::4} + if [ "$protocol" == "git@" ]; then + repo=${repo_url:${#protocol}}; + gitlab_url=${repo%%:*}; + else + protocol="https://" + repo=${repo_url:${#protocol}}; + gitlab_url=${repo%%/*}; + fi + + start=$((${#gitlab_url} + 1)) # +1 for the / (https) or : (ssh) + end=$((${#repo} - $start -4)) # -4 for the .git + repo=${repo:$start:$end} + repo=$(echo "$repo" | sed "s/\//%2F/g") + + # Send request + url="https://${gitlab_url}/api/v4/projects/${repo}/statuses/${bamboo_planRepository_revision}?state=${cistatus}&target_url=${bamboo_buildResultsUrl}" + echo "Sending request to $url" + curl --fail --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "$url" + ``` + ## Troubleshooting ### Builds not triggered diff --git a/doc/user/project/integrations/mock_ci.md b/doc/user/project/integrations/mock_ci.md index 631c53dcc44..5cde17dbd83 100644 --- a/doc/user/project/integrations/mock_ci.md +++ b/doc/user/project/integrations/mock_ci.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Mock CI Service **(FREE)** -**NB: This service is only listed if you are in a development environment!** +NOTE: +This service is only listed if you are in a [development environment](https://gitlab.com/gitlab-org/gitlab-mock-ci-service#setup-mockci-integration)! To set up the mock CI service server, respond to the following endpoints diff --git a/doc/user/project/integrations/webhook_events.md b/doc/user/project/integrations/webhook_events.md index ed62a34f6a3..32e5f949c15 100644 --- a/doc/user/project/integrations/webhook_events.md +++ b/doc/user/project/integrations/webhook_events.md @@ -824,6 +824,11 @@ The available values for `object_attributes.action` in the payload are: - `unapproval` - `merge` +The field `object_attributes.oldrev` is only available when there are actual code changes, like: + +- New code is pushed. +- A [suggestion](../merge_requests/reviews/suggestions.md) is applied. + Request header: ```plaintext @@ -868,6 +873,7 @@ Payload example: }, "object_attributes": { "id": 99, + "iid": 1, "target_branch": "master", "source_branch": "ms-viewport", "source_project_id": 14, @@ -879,10 +885,12 @@ Payload example: "milestone_id": null, "state": "opened", "blocking_discussions_resolved": true, + "work_in_progress": false, + "first_contribution": true, "merge_status": "unchecked", "target_project_id": 14, - "iid": 1, "description": "", + "url": "http://example.com/diaspora/merge_requests/1", "source": { "name":"Awesome Project", "description":"Aut reprehenderit ut est.", @@ -925,8 +933,18 @@ Payload example: "email": "gitlabdev@dv6700.(none)" } }, - "work_in_progress": false, - "url": "http://example.com/diaspora/merge_requests/1", + "labels": [{ + "id": 206, + "title": "API", + "color": "#ffffff", + "project_id": 14, + "created_at": "2013-12-03T17:15:43Z", + "updated_at": "2013-12-03T17:15:43Z", + "template": false, + "description": "API related issues", + "type": "ProjectLabel", + "group_id": 41 + }], "action": "open", "assignee": { "name": "User1", @@ -985,6 +1003,9 @@ Payload example: } ``` +NOTE: +The fields `assignee_id`, and `state` are deprecated. + ## Wiki page events Wiki page events are triggered when a wiki page is created, updated, or deleted. @@ -1147,6 +1168,9 @@ Payload example: "created_at": "2016-08-12 15:23:28 UTC", "started_at": null, "finished_at": null, + "duration": null, + "queued_duration": null, + "failure_reason": null, "when": "manual", "manual": true, "allow_failure": false, @@ -1175,7 +1199,10 @@ Payload example: "status": "success", "created_at": "2016-08-12 15:23:28 UTC", "started_at": "2016-08-12 15:26:12 UTC", - "finished_at": null, + "finished_at": "2016-08-12 15:26:29 UTC", + "duration": 17.0, + "queued_duration": 196.0, + "failure_reason": null, "when": "on_success", "manual": false, "allow_failure": false, @@ -1208,10 +1235,13 @@ Payload example: "id": 378, "stage": "test", "name": "test-build", - "status": "success", + "status": "failed", "created_at": "2016-08-12 15:23:28 UTC", "started_at": "2016-08-12 15:26:12 UTC", "finished_at": "2016-08-12 15:26:29 UTC", + "duration": 17.0, + "queued_duration": 196.0, + "failure_reason": "script_failure", "when": "on_success", "manual": false, "allow_failure": false, @@ -1247,6 +1277,9 @@ Payload example: "created_at": "2016-08-12 15:23:28 UTC", "started_at": "2016-08-12 15:24:56 UTC", "finished_at": "2016-08-12 15:25:26 UTC", + "duration": 17.0, + "queued_duration": 196.0, + "failure_reason": null, "when": "on_success", "manual": false, "allow_failure": false, @@ -1282,6 +1315,9 @@ Payload example: "created_at": "2016-08-12 15:23:28 UTC", "started_at": null, "finished_at": null, + "duration": null, + "queued_duration": null, + "failure_reason": null, "when": "on_success", "manual": false, "allow_failure": false, diff --git a/doc/user/project/issues/csv_import.md b/doc/user/project/issues/csv_import.md index 2fe3d78194c..1ae57c9a883 100644 --- a/doc/user/project/issues/csv_import.md +++ b/doc/user/project/issues/csv_import.md @@ -6,9 +6,20 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Importing issues from CSV **(FREE)** -Issues can be imported to a project by uploading a CSV file with the columns -`title` and `description`. Other columns are **not** imported. If you want to -retain columns such as labels and milestones, consider the [Move Issue feature](managing_issues.md#move-an-issue). +You can import issues to a project by uploading a CSV file with the following columns: + +| Name | Required? | Description | +|:--------------|:-----------------------|:-------------------------------------------------| +| `title` | **{check-circle}** Yes | Issue title. | +| `description` | **{check-circle}** Yes | Issue description. | +| `due_date` | **{dotted-circle}** No | Issue due date in `YYYY-MM-DD` format. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/91317) in GitLab 15.2. | + +Data in other columns is not imported. + +You can use the `description` field to embed [quick actions](../quick_actions.md) to add other data to the issue. +For example, labels, assignees, and milestones. + +Alternatively, you can [move an issue](managing_issues.md#move-an-issue). Moving issues preserves more data. The user uploading the CSV file is set as the author of the imported issues. @@ -44,16 +55,22 @@ To import issues, GitLab requires CSV files have a specific format: | double-quote character | The double-quote (`"`) character is used to quote fields, enabling the use of the column separator in a field (see the third line in the sample CSV data below). To insert a double-quote (`"`) in a quoted field use two double-quote characters in succession (`""`). | | data rows | After the header row, following rows must use the same column order. The issue title is required, but the description is optional. | -If you have special characters in a field, (such as `\n` or `,`), surround the -characters with double quotes (`"`). +If you have special characters (for example, `,` or `\n`) or multiple lines in a field (for example, +when using [quick actions](../quick_actions.md)), surround the characters with double quotes (`"`). + +When using [quick actions](../quick_actions.md), each action must be on a separate line. Sample CSV data: ```plaintext -title,description -My Issue Title,My Issue Description -Another Title,"A description, with a comma" -"One More Title","One More Description" +title,description,due date +My Issue Title,My Issue Description,2022-06-28 +Another Title,"A description, with a comma", +"One More Title","One More Description", +An Issue with Quick Actions,"Hey can we change the frontend? + +/assign @sjones +/label ~frontend ~documentation", ``` ### File size diff --git a/doc/user/project/issues/img/close_issue_from_board.gif b/doc/user/project/issues/img/close_issue_from_board.gif Binary files differdeleted file mode 100644 index 4814b42687b..00000000000 --- a/doc/user/project/issues/img/close_issue_from_board.gif +++ /dev/null diff --git a/doc/user/project/issues/img/multiple_assignees.gif b/doc/user/project/issues/img/multiple_assignees.gif Binary files differdeleted file mode 100644 index 055a0efd9ae..00000000000 --- a/doc/user/project/issues/img/multiple_assignees.gif +++ /dev/null diff --git a/doc/user/project/issues/img/turn_off_confidentiality_v15_0.png b/doc/user/project/issues/img/turn_off_confidentiality_v15_0.png Binary files differdeleted file mode 100644 index 37cbe0f4fd9..00000000000 --- a/doc/user/project/issues/img/turn_off_confidentiality_v15_0.png +++ /dev/null diff --git a/doc/user/project/issues/img/turn_on_confidentiality_v15_0.png b/doc/user/project/issues/img/turn_on_confidentiality_v15_0.png Binary files differdeleted file mode 100644 index 498867d1933..00000000000 --- a/doc/user/project/issues/img/turn_on_confidentiality_v15_0.png +++ /dev/null diff --git a/doc/user/project/issues/img/turn_on_confidentiality_v15_1.png b/doc/user/project/issues/img/turn_on_confidentiality_v15_1.png Binary files differindex 24a7ad554f8..c81ac85ab13 100644 --- a/doc/user/project/issues/img/turn_on_confidentiality_v15_1.png +++ b/doc/user/project/issues/img/turn_on_confidentiality_v15_1.png diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index fbdce211295..15d8da7f544 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -385,8 +385,6 @@ To close an issue, you can do the following: - At the top of the issue, select **Close issue**. - In an [issue board](../issue_board.md), drag an issue card from its list into the **Closed** list. - ![close issue from the issue board](img/close_issue_from_board.gif) - ### Reopen a closed issue Prerequisites: diff --git a/doc/user/project/issues/multiple_assignees_for_issues.md b/doc/user/project/issues/multiple_assignees_for_issues.md index 105dcd529c8..db160b6cfe8 100644 --- a/doc/user/project/issues/multiple_assignees_for_issues.md +++ b/doc/user/project/issues/multiple_assignees_for_issues.md @@ -4,39 +4,22 @@ group: Project Management info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Multiple Assignees for Issues **(PREMIUM)** +# Multiple assignees for issues **(PREMIUM)** -> Moved to GitLab Premium in 13.9. +> Moved from Starter to Premium in GitLab 13.9. -In large teams, where there is shared ownership of an issue, it can be difficult -to track who is working on it, who already completed their contributions, who -didn't even start yet. +In large teams with shared ownership, it can be difficult +to track who is working on an issue, who's already done, or who hasn't started yet. -You can also select multiple [assignees](managing_issues.md#assignee) for an issue, making it easier to +You can add multiple [assignees](managing_issues.md#assignee) to an issue, making it easier to track, and making clearer who is accountable for it. -![multiple assignees for issues](img/multiple_assignees_for_issues.png) - -## Use cases - -Consider a team formed by frontend developers, backend developers, -UX designers, QA testers, and a product manager working together to bring an idea to -market. - -Multiple Assignees for Issues makes collaboration smoother, +Multiple assignees for issues makes collaboration smoother, and allows shared responsibilities to be clearly displayed. All assignees are shown across your team's workflows and receive notifications (as they would as single assignees), simplifying communication and ownership. -Once an assignee had their work completed, they would remove themselves as assignees, making -it clear that their role is complete. +After an assignee completes their work, they remove themselves as an assignee, making +it clear that their task is complete. -## How it works - -From an opened issue, expand the right sidebar, locate the assignees entry, -and select **Edit**. From the dropdown menu, select as many users as you want -to assign the issue to. - -![adding multiple assignees](img/multiple_assignees.gif) - -To remove an assignee, clear them from the same dropdown menu. +![multiple assignees for issues](img/multiple_assignees_for_issues.png) diff --git a/doc/user/project/labels.md b/doc/user/project/labels.md index 160dade87bb..333b073ee40 100644 --- a/doc/user/project/labels.md +++ b/doc/user/project/labels.md @@ -441,8 +441,6 @@ This label now appears at the top of the label list, under **Prioritized Labels* To change the relative priority of these labels, drag them up and down the list. The labels higher in the list get higher priority. -![Drag to change label priority](img/labels_drag_priority_v12_1.gif) - To learn what happens when you sort by priority or label priority, see [Sorting and ordering issue lists](issues/sorting_issue_lists.md). diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index 7bea57d180b..ff5f2ac8cb6 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -158,7 +158,7 @@ group itself. Prerequisites: -- You must have the Owner role. +- You must have the Maintainer or Owner role. - Optional. Unassign the member from all issues and merge requests that are assigned to them. diff --git a/doc/user/project/members/share_project_with_groups.md b/doc/user/project/members/share_project_with_groups.md index 02a9b76ce38..c4ae00f3c6c 100644 --- a/doc/user/project/members/share_project_with_groups.md +++ b/doc/user/project/members/share_project_with_groups.md @@ -24,6 +24,13 @@ members. > - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) in GitLab 14.9. [Feature flag `invite_members_group_modal`](https://gitlab.com/gitlab-org/gitlab/-/issues/352526) removed. +You can share a project only with: + +- Groups for which you have an explicitly defined [membership](index.md). +- Groups that contain a nested subgroup or project for which you have an explicitly defined role. + +Administrators can share projects with any group in the namespace. + The primary mechanism to give a group of users, say 'Engineering', access to a project, say 'Project Acme', in GitLab is to make the 'Engineering' group the owner of 'Project Acme'. But what if 'Project Acme' already belongs to another group, say 'Open Source'? @@ -42,12 +49,11 @@ After sharing 'Project Acme' with 'Engineering': - The group is listed in the **Groups** tab. - The project is listed on the group dashboard. -You can share a project only with: - -- Groups for which you have an explicitly defined membership. -- Groups that contain a nested subgroup or project for which you have an explicitly defined role. +When you share a project, be aware of the following restrictions and outcomes: -Administrators can share projects with any group in the system. +- [Maximum access level](#maximum-access-level) +- [Sharing a public project with a private group](#share-a-public-project-with-private-group) +- [Sharing project with group lock](#share-project-with-group-lock) ## Maximum access level @@ -61,9 +67,13 @@ in. That means you can only share down the hierarchy. For example, `group/subgro - Can not be shared with `group`. - Can be shared with `group/subgroup02` or `group/subgroup01/subgroup03`. -## Share public project with private group +## Share a public project with private group + +When you share a public project with a private group, be aware of the following outcomes: -When sharing a public project with a private group, owners and maintainers of the project see the name of the group in the `members` page. Owners also have the possibility to see members of the private group they don't have access to when mentioning them in the issue or merge request. +- The name of the group is no longer private and is visible to all users in the project members page. +- Owners of the project have access to members of the private group when they mention them in issues or merge requests. +- Project members who are direct or indirect members of the private group can see private group members listed in addition to members of the project. ## Share project with group lock diff --git a/doc/user/project/merge_requests/accessibility_testing.md b/doc/user/project/merge_requests/accessibility_testing.md index b8907532066..c1a87f7a5d4 100644 --- a/doc/user/project/merge_requests/accessibility_testing.md +++ b/doc/user/project/merge_requests/accessibility_testing.md @@ -1,76 +1,11 @@ --- -stage: Verify -group: Pipeline Insights -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../../ci/testing/accessibility_testing.md' +remove_date: '2022-08-31' --- -# Accessibility testing **(FREE)** +This document was moved to [another location](../../../ci/testing/accessibility_testing.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25144) in GitLab 12.8. - -If your application offers a web interface, you can use -[GitLab CI/CD](../../../ci/index.md) to determine the accessibility -impact of pending code changes. - -[Pa11y](https://pa11y.org/) is a free and open source tool for -measuring the accessibility of web sites. GitLab integrates Pa11y into a -[CI job template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml). -The `a11y` job analyzes a defined set of web pages and reports -accessibility violations, warnings, and notices in a file named -`accessibility`. - -As of [GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73309), Pa11y uses -[WCAG 2.1 rules](https://www.w3.org/TR/WCAG21/#new-features-in-wcag-2-1). - -## Accessibility merge request widget - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39425) in GitLab 13.0 behind the disabled [feature flag](../../../administration/feature_flags.md) `:accessibility_report_view`. -> - [Feature Flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/217372) in GitLab 13.1. - -GitLab displays an **Accessibility Report** in the merge request widget area: - -![Accessibility merge request widget](img/accessibility_mr_widget_v13_0.png) - -## Configure accessibility testing - -You can run Pa11y with GitLab CI/CD using the -[GitLab Accessibility Docker image](https://gitlab.com/gitlab-org/ci-cd/accessibility). - -To define the `a11y` job for GitLab 12.9 and later: - -1. [Include](../../../ci/yaml/index.md#includetemplate) the - [`Accessibility.gitlab-ci.yml` template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml) - from your GitLab installation. -1. Add the following configuration to your `.gitlab-ci.yml` file. - - ```yaml - stages: - - accessibility - - variables: - a11y_urls: "https://about.gitlab.com https://gitlab.com/users/sign_in" - - include: - - template: "Verify/Accessibility.gitlab-ci.yml" - ``` - -1. Customize the `a11y_urls` variable to list the URLs of the web pages to test with Pa11y. - -The `a11y` job in your CI/CD pipeline generates these files: - -- One HTML report per URL listed in the `a11y_urls` variable. -- One file containing the collected report data. In GitLab versions 12.11 and later, this - file is named `gl-accessibility.json`. In GitLab versions 12.10 and earlier, this file - is named [`accessibility.json`](https://gitlab.com/gitlab-org/ci-cd/accessibility/-/merge_requests/9). - -You can [view job artifacts in your browser](../../../ci/pipelines/job_artifacts.md#download-job-artifacts). - -NOTE: -For GitLab versions earlier than 12.9, use `include:remote` and -link to the [current template in the default branch](https://gitlab.com/gitlab-org/gitlab/-/raw/master/lib/gitlab/ci/templates/Verify/Accessibility.gitlab-ci.yml) - -NOTE: -The job definition provided by the template does not support Kubernetes. - -You cannot pass configurations into Pa11y via CI configuration. -To change the configuration, edit a copy of the template in your CI file. +<!-- This redirect file can be deleted after <2022-09-22>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/merge_requests/approvals/img/mr_approvals_by_code_owners_v12_7.png b/doc/user/project/merge_requests/approvals/img/mr_approvals_by_code_owners_v12_7.png Binary files differdeleted file mode 100644 index 669148a41d8..00000000000 --- a/doc/user/project/merge_requests/approvals/img/mr_approvals_by_code_owners_v12_7.png +++ /dev/null diff --git a/doc/user/project/merge_requests/approvals/img/mr_approvals_by_code_owners_v15_2.png b/doc/user/project/merge_requests/approvals/img/mr_approvals_by_code_owners_v15_2.png Binary files differnew file mode 100644 index 00000000000..37dad4e5ae8 --- /dev/null +++ b/doc/user/project/merge_requests/approvals/img/mr_approvals_by_code_owners_v15_2.png diff --git a/doc/user/project/merge_requests/approvals/rules.md b/doc/user/project/merge_requests/approvals/rules.md index 21cf5cca4d1..b79c8ee867f 100644 --- a/doc/user/project/merge_requests/approvals/rules.md +++ b/doc/user/project/merge_requests/approvals/rules.md @@ -152,9 +152,9 @@ become eligible approvers in the project. To enable this merge request approval 1. Go to your project and select **Settings > General**. 1. Expand **Merge request (MR) approvals**. -1. Locate **Eligible users** and select the number of approvals required: +1. Locate **All eligible users** and select the number of approvals required: - ![MR approvals by Code Owners](img/mr_approvals_by_code_owners_v12_7.png) +![MR approvals by Code Owners](img/mr_approvals_by_code_owners_v15_2.png) You can also [require code owner approval](../../protected_branches.md#require-code-owner-approval-on-a-protected-branch) diff --git a/doc/user/project/merge_requests/approvals/settings.md b/doc/user/project/merge_requests/approvals/settings.md index 9295ea4c310..7b865a91106 100644 --- a/doc/user/project/merge_requests/approvals/settings.md +++ b/doc/user/project/merge_requests/approvals/settings.md @@ -55,7 +55,7 @@ this setting, unless you configure one of these options: > Moved to GitLab Premium in 13.9. By default, users who commit to a merge request can still approve it. At both -the project level or [instance level](../../../admin_area/merge_requests_approvals.md) +the project level or [instance level](../../../admin_area/merge_requests_approvals.md), you can prevent committers from approving merge requests that are partially their own. To do this: @@ -82,7 +82,7 @@ read the official Git documentation for an explanation. ## Prevent editing approval rules in merge requests By default, users can override the approval rules you [create for a project](rules.md) -on a per-merge request basis. If you don't want users to change approval rules +on a per-merge-request basis. If you don't want users to change approval rules on merge requests, you can disable this setting: 1. Go to your project and select **Settings > General**. @@ -119,7 +119,7 @@ when more changes are added to it: 1. Select the **Remove all approvals when commits are added to the source branch** checkbox. 1. Select **Save changes**. -Approvals aren't reset when a merge request is [rebased from the UI](../methods/index.md#rebasing-in-semi-linear-merge-methods) +Approvals aren't reset when a merge request is [rebased from the UI](../methods/index.md#rebasing-in-semi-linear-merge-methods). However, approvals are reset if the target branch is changed. ## Code coverage check approvals diff --git a/doc/user/project/merge_requests/browser_performance_testing.md b/doc/user/project/merge_requests/browser_performance_testing.md index 9c7d9e2bf19..95f749210c4 100644 --- a/doc/user/project/merge_requests/browser_performance_testing.md +++ b/doc/user/project/merge_requests/browser_performance_testing.md @@ -1,242 +1,11 @@ --- -stage: Verify -group: Pipeline Insights -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../../ci/testing/browser_performance_testing.md' +remove_date: '2022-08-31' --- -# Browser Performance Testing **(PREMIUM)** +This document was moved to [another location](../../../ci/testing/browser_performance_testing.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3507) in GitLab 10.3. - -If your application offers a web interface and you're using -[GitLab CI/CD](../../../ci/index.md), you can quickly determine the rendering performance -impact of pending code changes in the browser. - -NOTE: -You can automate this feature in your applications by using [Auto DevOps](../../../topics/autodevops/index.md). - -## Overview - -GitLab uses [Sitespeed.io](https://www.sitespeed.io), a free and open source -tool, for measuring the rendering performance of web sites. The -[Sitespeed plugin](https://gitlab.com/gitlab-org/gl-performance) that GitLab built outputs -the performance score for each page analyzed in a file called `browser-performance.json` -this data can be shown on Merge Requests. - -## Use cases - -Consider the following workflow: - -1. A member of the marketing team is attempting to track engagement by adding a new tool. -1. With browser performance metrics, they see how their changes are impacting the usability - of the page for end users. -1. The metrics show that after their changes, the performance score of the page has gone down. -1. When looking at the detailed report, they see the new JavaScript library was - included in `<head>`, which affects loading page speed. -1. They ask for help from a front end developer, who sets the library to load asynchronously. -1. The frontend developer approves the merge request, and authorizes its deployment to production. - -## How browser performance testing works - -First, define a job in your `.gitlab-ci.yml` file that generates the -[Browser Performance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsbrowser_performance). -GitLab then checks this report, compares key performance metrics for each page -between the source and target branches, and shows the information in the merge request. - -For an example Browser Performance job, see -[Configuring Browser Performance Testing](#configuring-browser-performance-testing). - -NOTE: -If the Browser Performance report has no data to compare, such as when you add the -Browser Performance job in your `.gitlab-ci.yml` for the very first time, -the Browser Performance report widget doesn't display. It must have run at least -once on the target branch (`main`, for example), before it displays in a -merge request targeting that branch. - -![Browser Performance Widget](img/browser_performance_testing.png) - -## Configuring Browser Performance Testing - -This example shows how to run the [sitespeed.io container](https://hub.docker.com/r/sitespeedio/sitespeed.io/) -on your code by using GitLab CI/CD and [sitespeed.io](https://www.sitespeed.io) -using Docker-in-Docker. - -1. First, set up GitLab Runner with a - [Docker-in-Docker build](../../../ci/docker/using_docker_build.md#use-docker-in-docker). -1. Configure the default Browser Performance Testing CI/CD job as follows in your `.gitlab-ci.yml` file: - - ```yaml - include: - template: Verify/Browser-Performance.gitlab-ci.yml - - browser_performance: - variables: - URL: https://example.com - ``` - -WARNING: -In GitLab 13.12 and earlier, the job [was named](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) `performance`. - -The above example: - -- Creates a `browser_performance` job in your CI/CD pipeline and runs sitespeed.io against the webpage you - defined in `URL` to gather key metrics. -- Uses a template that doesn't work with Kubernetes clusters. If you are using a Kubernetes cluster, - use [`template: Jobs/Browser-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Browser-Performance-Testing.gitlab-ci.yml) - instead. -- Uses a CI/CD template that is included in all GitLab installations since 12.4. If you are using - GitLab 12.3 or earlier, you must [add the configuration manually](#gitlab-versions-132-and-earlier). - -The template uses the [GitLab plugin for sitespeed.io](https://gitlab.com/gitlab-org/gl-performance), -and it saves the full HTML sitespeed.io report as a [Browser Performance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsbrowser_performance) -that you can later download and analyze. This implementation always takes the latest -Browser Performance artifact available. If [GitLab Pages](../pages/index.md) is enabled, -you can view the report directly in your browser. - -You can also customize the jobs with CI/CD variables: - -- `SITESPEED_IMAGE`: Configure the Docker image to use for the job (default `sitespeedio/sitespeed.io`), but not the image version. -- `SITESPEED_VERSION`: Configure the version of the Docker image to use for the job (default `14.1.0`). -- `SITESPEED_OPTIONS`: Configure any additional sitespeed.io options as required (default `nil`). Refer to the [sitespeed.io documentation](https://www.sitespeed.io/documentation/sitespeed.io/configuration/) for more details. - -For example, you can override the number of runs sitespeed.io -makes on the given URL, and change the version: - -```yaml -include: - template: Verify/Browser-Performance.gitlab-ci.yml - -browser_performance: - variables: - URL: https://www.sitespeed.io/ - SITESPEED_VERSION: 13.2.0 - SITESPEED_OPTIONS: -n 5 -``` - -### Configuring degradation threshold - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27599) in GitLab 13.0. - -You can configure the sensitivity of degradation alerts to avoid getting alerts for minor drops in metrics. -This is done by setting the `DEGRADATION_THRESHOLD` CI/CD variable. In the example below, the alert only shows up -if the `Total Score` metric degrades by 5 points or more: - -```yaml -include: - template: Verify/Browser-Performance.gitlab-ci.yml - -browser_performance: - variables: - URL: https://example.com - DEGRADATION_THRESHOLD: 5 -``` - -The `Total Score` metric is based on sitespeed.io's [coach performance score](https://www.sitespeed.io/documentation/sitespeed.io/metrics/#performance-score). There is more information in [the coach documentation](https://www.sitespeed.io/documentation/coach/how-to/#what-do-the-coach-do). - -### Performance testing on Review Apps - -The above CI YAML configuration is great for testing against static environments, and it can -be extended for dynamic environments, but a few extra steps are required: - -1. The `browser_performance` job should run after the dynamic environment has started. -1. In the `review` job: - 1. Generate a URL list file with the dynamic URL. - 1. Save the file as an artifact, for example with `echo $CI_ENVIRONMENT_URL > environment_url.txt` - in your job's `script`. - 1. Pass the list as the URL environment variable (which can be a URL or a file containing URLs) - to the `browser_performance` job. -1. You can now run the sitespeed.io container against the desired hostname and - paths. - -Your `.gitlab-ci.yml` file would look like: - -```yaml -stages: - - deploy - - performance - -include: - template: Verify/Browser-Performance.gitlab-ci.yml - -review: - stage: deploy - environment: - name: review/$CI_COMMIT_REF_SLUG - url: http://$CI_COMMIT_REF_SLUG.$APPS_DOMAIN - script: - - run_deploy_script - - echo $CI_ENVIRONMENT_URL > environment_url.txt - artifacts: - paths: - - environment_url.txt - only: - - branches - except: - - master - -browser_performance: - dependencies: - - review - variables: - URL: environment_url.txt -``` - -### GitLab versions 13.2 and earlier - -Browser Performance Testing has gone through several changes since its introduction. -In this section we detail these changes and how you can run the test based on your -GitLab version: - -- In 13.2 the feature was renamed from `Performance` to `Browser Performance` with additional - template CI/CD variables. -- In GitLab 12.4 [a job template was made available](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Browser-Performance.gitlab-ci.yml). -- For 11.5 to 12.3 no template is available and the job has to be defined manually as follows: - - ```yaml - performance: - stage: performance - image: docker:git - variables: - URL: https://example.com - SITESPEED_VERSION: 14.1.0 - SITESPEED_OPTIONS: '' - services: - - docker:stable-dind - script: - - mkdir gitlab-exporter - - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/1.1.0/index.js - - mkdir sitespeed-results - - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:$SITESPEED_VERSION --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL $SITESPEED_OPTIONS - - mv sitespeed-results/data/performance.json performance.json - artifacts: - paths: - - performance.json - - sitespeed-results/ - reports: - performance: performance.json - ``` - -- For 11.4 and earlier the job should be defined as follows: - - ```yaml - performance: - stage: performance - image: docker:git - variables: - URL: https://example.com - services: - - docker:stable-dind - script: - - mkdir gitlab-exporter - - wget -O ./gitlab-exporter/index.js https://gitlab.com/gitlab-org/gl-performance/raw/1.1.0/index.js - - mkdir sitespeed-results - - docker run --shm-size=1g --rm -v "$(pwd)":/sitespeed.io sitespeedio/sitespeed.io:6.3.1 --plugins.add ./gitlab-exporter --outputFolder sitespeed-results $URL - - mv sitespeed-results/data/performance.json performance.json - artifacts: - paths: - - performance.json - - sitespeed-results/ - ``` - -Upgrading to the latest version and using the templates is recommended, to ensure -you receive the latest updates, including updates to the sitespeed.io versions. +<!-- This redirect file can be deleted after <2022-09-22>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/merge_requests/code_quality.md b/doc/user/project/merge_requests/code_quality.md index 623af914692..79e590cb905 100644 --- a/doc/user/project/merge_requests/code_quality.md +++ b/doc/user/project/merge_requests/code_quality.md @@ -1,634 +1,11 @@ --- -stage: Secure -group: Static Analysis -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../../ci/testing/code_quality.md' +remove_date: '2022-08-31' --- -# Code Quality **(FREE)** +This document was moved to [another location](../../../ci/testing/code_quality.md). -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. - -To ensure your project's code stays simple, readable, and easy to contribute to, -you can use [GitLab CI/CD](../../../ci/index.md) to analyze your source code quality. - -For example, while you're implementing a feature, you can run Code Quality reports -to analyze how your improvements are impacting your code's quality. You can -use this information to ensure that your changes are improving performance rather -than degrading it. - -Code Quality: - -- Uses [plugins](https://docs.codeclimate.com/docs/list-of-engines) supported by Code Climate, which are - free and open source. Code Quality does not require a Code Climate - subscription. -- Runs in [pipelines](../../../ci/pipelines/index.md) by using a Docker image built in the - [GitLab Code Quality](https://gitlab.com/gitlab-org/ci-cd/codequality) project. -- Uses [default Code Climate configurations](https://gitlab.com/gitlab-org/ci-cd/codequality/-/tree/master/codeclimate_defaults). -- Can make use of a [template](#example-configuration). -- Is available by using [Auto Code Quality](../../../topics/autodevops/stages.md#auto-code-quality), provided by [Auto DevOps](../../../topics/autodevops/index.md). -- Can be extended through [Analysis Plugins](https://docs.codeclimate.com/docs/list-of-engines) or a [custom tool](#implementing-a-custom-tool). - -## Summary of features per tier - -Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), -as shown in the following table: - -| Capability | In Free | In Premium | In Ultimate | -|:----------------------------------------------------------------------|:--------------------|:--------------------|:-------------------| -| [Configure scanners](#configuring-jobs-using-variables) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [Integrate custom scanners](#implementing-a-custom-tool) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [Generate JSON or HTML report artifacts](#generate-an-html-report) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [See findings in merge request widget](#code-quality-widget) | **{check-circle}** | **{check-circle}** | **{check-circle}** | -| [See reports in CI pipelines](#code-quality-reports) | **{dotted-circle}** | **{check-circle}** | **{check-circle}** | -| [See findings in merge request diff view](#code-quality-in-diff-view) | **{dotted-circle}** | **{dotted-circle}** | **{check-circle}** | - -## Code Quality Widget - -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. - -Going a step further, GitLab can show the Code Quality report right -in the merge request widget area if a report from the target branch is available to compare to: - -![Code Quality Widget](img/code_quality_widget_13_11.png) - -Watch a quick walkthrough of Code Quality in action: - -<div class="video-fallback"> - See the video: <a href="https://www.youtube.com/watch?v=B32LxtJKo9M">Code Quality: Speed Run</a>. -</div> -<figure class="video-container"> - <iframe src="https://www.youtube.com/embed/B32LxtJKo9M" frameborder="0" allowfullscreen="true"> </iframe> -</figure> - -NOTE: -For one customer, the auditor found that having Code Quality, SAST, and Container Scanning all automated in GitLab CI/CD was almost better than a manual review! [Read more](https://about.gitlab.com/customers/bi_worldwide/). - -See also the Code Climate list of [Supported Languages for Maintainability](https://docs.codeclimate.com/docs/supported-languages-for-maintainability). - -## Code Quality in diff view **(ULTIMATE)** - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267612) in GitLab 13.11, disabled by default behind the `codequality_mr_diff` [feature flag](../../../administration/feature_flags.md). -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 13.12. -> - [Disabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) in GitLab 14.0 due to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/334116). -> - [Inline annotation added](https://gitlab.com/gitlab-org/gitlab/-/issues/2526) and [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284140) in GitLab 14.1. - -Changes to files in merge requests can cause Code Quality to fall if merged. In these cases, -the merge request's diff view displays an indicator next to lines with new Code Quality violations. For example: - -![Code Quality MR diff report](img/code_quality_mr_diff_report_v14_2.png) - -## Example configuration - -This example shows how to run Code Quality on your code by using GitLab CI/CD and Docker. - -- Using shared runners, the job should be configured For the [Docker-in-Docker workflow](../../../ci/docker/using_docker_build.md#use-docker-in-docker). -- Using private runners, there is an [alternative configuration](#set-up-a-private-runner-for-code-quality-without-docker-in-docker) recommended for running Code Quality analysis more efficiently. - -In either configuration, the runner must have enough disk space to handle generated Code Quality files. For example on the [GitLab project](https://gitlab.com/gitlab-org/gitlab) the files are approximately 7 GB. - -Once you set up GitLab Runner, include the [Code Quality template](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Code-Quality.gitlab-ci.yml) in your CI configuration: - -```yaml -include: - - template: Code-Quality.gitlab-ci.yml -``` - -The above example creates a `code_quality` job in your CI/CD pipeline which -scans your source code for code quality issues. The report is saved as a -[Code Quality report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportscodequality) -that you can later download and analyze. - -It's also possible to override the URL to the Code Quality image by -setting the `CODE_QUALITY_IMAGE` CI/CD variable. This is particularly useful if you want -to lock in a specific version of Code Quality, or use a fork of it: - -```yaml -include: - - template: Code-Quality.gitlab-ci.yml - -code_quality: - variables: - CODE_QUALITY_IMAGE: "registry.example.com/codequality-fork:latest" -``` - -In [GitLab 13.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/11100), you can override the [Code Quality environment variables](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables): - -```yaml -variables: - TIMEOUT_SECONDS: 1 - -include: - - template: Code-Quality.gitlab-ci.yml -``` - -By default, report artifacts are not downloadable. If you need them downloadable on the -job details page, you can add `gl-code-quality-report.json` to the artifact paths like so: - -```yaml -include: - - template: Code-Quality.gitlab-ci.yml - -code_quality: - artifacts: - paths: [gl-code-quality-report.json] -``` - -The included `code_quality` job is running in the `test` stage, so it needs to be included in your CI configuration, like so: - -```yaml -stages: - - test -``` - -NOTE: -This information is automatically extracted and shown right in the merge request widget. - -WARNING: -On self-managed instances, if a malicious actor compromises the Code Quality job -definition they could execute privileged Docker commands on the runner -host. Having proper access control policies mitigates this attack vector by -allowing access only to trusted actors. - -### Set up a private runner for code quality without Docker-in-Docker - -It's possible to configure your own runners and avoid Docker-in-Docker. You can use a -configuration that may greatly speed up job execution without requiring your runners -to operate in privileged mode. - -This alternative configuration uses socket binding to share the Runner's Docker daemon -with the job environment. Be aware that this configuration [has significant considerations](../../../ci/docker/using_docker_build.md#use-docker-socket-binding) -to be consider, but may be preferable depending on your use case. - -1. Register a new runner: - - ```shell - $ gitlab-runner register --executor "docker" \ - --docker-image="docker:stable" \ - --url "https://gitlab.com/" \ - --description "cq-sans-dind" \ - --tag-list "cq-sans-dind" \ - --locked="false" \ - --access-level="not_protected" \ - --docker-volumes "/cache"\ - --docker-volumes "/builds:/builds"\ - --docker-volumes "/var/run/docker.sock:/var/run/docker.sock" \ - --registration-token="<project_token>" \ - --non-interactive - ``` - -1. **Optional, but recommended:** Set the builds directory to `/tmp/builds`, - so job artifacts are periodically purged from the runner host. If you skip - this step, you must clean up the default builds directory (`/builds`) yourself. - You can do this by adding the following two flags to `gitlab-runner register` - in the previous step. - - ```shell - --builds-dir "/tmp/builds" - --docker-volumes "/tmp/builds:/tmp/builds" # Use this instead of --docker-volumes "/builds:/builds" - ``` - - The resulting configuration: - - ```toml - [[runners]] - name = "cq-sans-dind" - url = "https://gitlab.com/" - token = "<project_token>" - executor = "docker" - builds_dir = "/tmp/builds" - [runners.docker] - tls_verify = false - image = "docker:stable" - privileged = false - disable_entrypoint_overwrite = false - oom_kill_disable = false - disable_cache = false - volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock", "/tmp/builds:/tmp/builds"] - shm_size = 0 - [runners.cache] - [runners.cache.s3] - [runners.cache.gcs] - ``` - -1. Apply two overrides to the `code_quality` job created by the template: - - ```yaml - include: - - template: Code-Quality.gitlab-ci.yml - - code_quality: - services: # Shut off Docker-in-Docker - tags: - - cq-sans-dind # Set this job to only run on our new specialized runner - ``` - -The end result is that: - -- Privileged mode is not used. -- Docker-in-Docker is not used. -- Docker images, including all CodeClimate images, are cached, and not re-fetched for subsequent jobs. - -With this configuration, the run time for a second pipeline is much shorter. For example -this [small change](https://gitlab.com/drew/test-code-quality-template/-/merge_requests/4/diffs?commit_id=1e705607aef7236c1b20bb6f637965f3f3e53a46) -to an [open merge request](https://gitlab.com/drew/test-code-quality-template/-/merge_requests/4/pipelines) -running Code Quality analysis ran significantly faster the second time: - -![Code Quality sequential runs without DinD](img/code_quality_host_bound_sequential.png) - -This configuration is not possible on `gitlab.com` shared runners. Shared runners -are configured with `privileged=true`, and they do not expose `docker.sock` into -the job container. As a result, socket binding cannot be used to make `docker` available -in the context of the job script. - -[Docker-in-Docker](../../../ci/docker/using_docker_build.md#use-docker-in-docker) -was chosen as an operational decision by the runner team, instead of exposing `docker.sock`. - -### Disabling the code quality job - -The `code_quality` job doesn't run if the `$CODE_QUALITY_DISABLED` CI/CD variable -is present. Please refer to the CI/CD variables [documentation](../../../ci/variables/index.md) -to learn more about how to define one. - -To disable the `code_quality` job, add `CODE_QUALITY_DISABLED` as a custom CI/CD variable. -This can be done: - -- For [the whole project](../../../ci/variables/index.md#custom-cicd-variables). -- For a single pipeline run: - - 1. Go to **CI/CD > Pipelines** - 1. Select **Run pipeline** - 1. Add `CODE_QUALITY_DISABLED` as the variable key, with any value. - -### Using with merge request pipelines - -The configuration provided by the Code Quality template does not let the `code_quality` job -run on [merge request pipelines](../../../ci/pipelines/merge_request_pipelines.md). - -If merge request pipelines is enabled, the `code_quality:rules` must be redefined. - -The template has these [`rules`](../../../ci/yaml/index.md#rules) for the `code quality` job: - -```yaml -code_quality: - rules: - - if: $CODE_QUALITY_DISABLED - when: never - - if: $CI_COMMIT_TAG || $CI_COMMIT_BRANCH -``` - -If you are using merge request pipelines, your `rules` (or [`workflow: rules`](../../../ci/yaml/index.md#workflow)) -might look like this example: - -```yaml -job1: - rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" # Run job1 in merge request pipelines - - if: $CI_COMMIT_BRANCH == "main" # Run job1 in pipelines on the main branch (but not in other branch pipelines) - - if: $CI_COMMIT_TAG # Run job1 in pipelines for tags -``` - -To make these work together, you need to overwrite the code quality `rules` -so that they match your current `rules`. From the example above, it could look like: - -```yaml -include: - - template: Code-Quality.gitlab-ci.yml - -code_quality: - rules: - - if: $CODE_QUALITY_DISABLED - when: never - - if: $CI_PIPELINE_SOURCE == "merge_request_event" # Run code quality job in merge request pipelines - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run code quality job in pipelines on the default branch (but not in other branch pipelines) - - if: $CI_COMMIT_TAG # Run code quality job in pipelines for tags -``` - -### Configure Code Quality to use a private container image registry - -> [Introduced](https://gitlab.com/gitlab-org/ci-cd/codequality/-/merge_requests/30) in 13.7. - -To reduce network time and external dependency, you can use your own -container image registry to host the Code Quality Docker images. Because of -the nested architecture of container execution, the registry prefix must -be specifically configured to be passed down into CodeClimate's subsequent -`docker pull` commands for individual engines. - -The following two variables can address all of the required image pulls: - -- `CODE_QUALITY_IMAGE`: A fully prefixed image name that can be located anywhere - accessible from your job environment. GitLab Container Registry can be used here - to host your own copy. -- `CODECLIMATE_PREFIX`: The domain of your intended container image registry. This - is a configuration option supported by [CodeClimate CLI](https://github.com/codeclimate/codeclimate/pull/948). You must: - - Include a trailing slash (`/`). - - Not include a protocol prefix, such as `https://`. - -```yaml -include: - - template: Jobs/Code-Quality.gitlab-ci.yml - -code_quality: - variables: - CODE_QUALITY_IMAGE: "my-private-registry.local:12345/codequality:0.85.24" - CODECLIMATE_PREFIX: "my-private-registry.local:12345/" -``` - -This example is specific to GitLab Code Quality. For more general -instructions on how to configure DinD with a registry mirror, see the -relevant [documentation](../../../ci/docker/using_docker_build.md#enable-registry-mirror-for-dockerdind-service). - -## Configuring jobs using variables - -The Code Quality job supports environment variables that users can set to -configure job execution at runtime. - -For a list of available environment variables, see -[Environment variables](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables). - -## Implementing a custom tool - -It's possible to have a custom tool provide Code Quality reports in GitLab. To -do this: - -1. Define a job in your `.gitlab-ci.yml` file that generates the - [Code Quality report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportscodequality). -1. Configure your tool to generate the Code Quality report artifact as a JSON - file that implements a subset of the [Code Climate - spec](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types). - -The Code Quality report artifact JSON file must contain an array of objects -with the following properties: - -| Name | Description | -| ---------------------- | ----------------------------------------------------------------------------------------- | -| `description` | A description of the code quality violation. | -| `fingerprint` | A unique fingerprint to identify the code quality violation. For example, an MD5 hash. | -| `severity` | A severity string (can be `info`, `minor`, `major`, `critical`, or `blocker`). | -| `location.path` | The relative path to the file containing the code quality violation. | -| `location.lines.begin` or `location.positions.begin.line` | The line on which the code quality violation occurred. | - -Example: - -```json -[ - { - "description": "'unused' is assigned a value but never used.", - "fingerprint": "7815696ecbf1c96e6894b779456d330e", - "severity": "minor", - "location": { - "path": "lib/index.js", - "lines": { - "begin": 42 - } - } - } -] -``` - -NOTE: -Although the Code Climate spec supports more properties, those are ignored by -GitLab. -The GitLab parser does not allow a [byte order mark](https://en.wikipedia.org/wiki/Byte_order_mark) -at the beginning of the file. - -## Code Quality reports **(PREMIUM)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21527) in GitLab 12.9. - -![Code Quality Report](img/code_quality_report_13_11.png) - -After the Code Quality job completes: - -- Potential changes to code quality are shown directly in the merge request. - The Code Quality widget in the merge request compares the reports from the base and head of the branch, - then lists any violations that are resolved or created when the branch is merged. -- The full JSON report is available as a - [downloadable artifact](../../../ci/pipelines/job_artifacts.md#download-job-artifacts) - for the `code_quality` job. -- The full list of code quality violations generated by a pipeline is shown in the - Code Quality tab of the Pipeline Details page. - -## Generate an HTML report - -In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/ci-cd/codequality/-/issues/10), -it is possible to generate an HTML report file by setting the `REPORT_FORMAT` -CI/CD variable to `html`. This is useful if you just want to view the report in a more -human-readable format or to publish this artifact on GitLab Pages for even -easier reviewing. - -To generate both JSON and HTML report files, add another job to your template by using `extends: code_quality`: - -```yaml -include: - - template: Code-Quality.gitlab-ci.yml - -code_quality_html: - extends: code_quality - variables: - REPORT_FORMAT: html - artifacts: - paths: [gl-code-quality-report.html] -``` - -NOTE: -Adding a job means your code is scanned twice: once to generate a JSON report and once to generate an HTML report. - -You can also generate _only_ an HTML report instead of the standard JSON report. To do so, set `REPORT_FORMAT` to `html` in the existing job: - -```yaml -include: - - template: Code-Quality.gitlab-ci.yml - -code_quality: - variables: - REPORT_FORMAT: html - artifacts: - paths: [gl-code-quality-report.html] -``` - -WARNING: -If you only generate an HTML report, you can't see your results in the [merge request widget](#code-quality-widget), [pipeline report](#code-quality-reports), or [diff view](#code-quality-in-diff-view). -These features require a JSON report. - -## Extending functionality - -### Using Analysis Plugins - -Should there be a need to extend the default functionality provided by Code Quality, as stated in [Code Quality](#code-quality), [Analysis Plugins](https://docs.codeclimate.com/docs/list-of-engines) are available. - -For example, to use the [SonarJava analyzer](https://docs.codeclimate.com/docs/sonar-java), -add a file named `.codeclimate.yml` containing the [enablement code](https://docs.codeclimate.com/docs/sonar-java#enable-the-plugin) -for the plugin to the root of your repository: - -```yaml -version: "2" -plugins: - sonar-java: - enabled: true -``` - -This adds SonarJava to the `plugins:` section of the [default `.codeclimate.yml`](https://gitlab.com/gitlab-org/ci-cd/codequality/-/blob/master/codeclimate_defaults/.codeclimate.yml.template) -included in your project. - -Changes to the `plugins:` section do not affect the `exclude_patterns` section of the -default `.codeclimate.yml`. See the Code Climate documentation for -[excluding files and folders](https://docs.codeclimate.com/docs/excluding-files-and-folders) -for more details. - -Here's [an example project](https://gitlab.com/jheimbuck_gl/jh_java_example_project) that uses Code Quality with a `.codeclimate.yml` file. - -## Use a Code Quality image hosted in a registry with untrusted certificates - -If you set the `CODE_QUALITY_IMAGE` to an image that is hosted in a -Docker registry which uses a TLS certificate that is not trusted, such as -a self-signed certificate, you can see errors like the one below: - -```shell -$ docker pull --quiet "$CODE_QUALITY_IMAGE" -Error response from daemon: Get https://gitlab.example.com/v2/: x509: certificate signed by unknown authority -``` - -To fix this, configure the Docker daemon to [trust certificates](https://docs.docker.com/registry/insecure/#use-self-signed-certificates) -by putting the certificate inside of the `/etc/docker/certs.d` -directory. - -This Docker daemon is exposed to the subsequent Code Quality Docker container in the -[GitLab Code Quality template](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.8.3-ee/lib/gitlab/ci/templates/Jobs/Code-Quality.gitlab-ci.yml#L41) -and should be to exposed any other containers in which you want to have -your certificate configuration apply. - -### Docker - -If you have access to GitLab Runner configuration, add the directory as a -[volume mount](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#volumes-in-the-runnersdocker-section). For example: - -```toml -[[runners]] - ... - executor = "docker" - [runners.docker] - ... - privileged = true - volumes = ["/cache", "/etc/gitlab-runner/certs/gitlab.example.com.crt:/etc/docker/certs.d/gitlab.example.com/ca.crt:ro"] -``` - -Replace `gitlab.example.com` with the actual domain of the registry. - -### Kubernetes - -If you have access to GitLab Runner configuration and the Kubernetes cluster, -you can [mount a ConfigMap](https://docs.gitlab.com/runner/executors/kubernetes.html#configmap-volumes): - -1. Create a ConfigMap with the certificate: - - ```shell - kubectl create configmap registry-crt --namespace gitlab-runner --from-file /etc/gitlab-runner/certs/gitlab.example.com.crt - ``` - -1. Update GitLab Runner `config.toml` to specify the ConfigMap: - - ```toml - [[runners]] - ... - executor = "kubernetes" - [runners.kubernetes] - image = "alpine:3.12" - privileged = true - [[runners.kubernetes.volumes.config_map]] - name = "registry-crt" - mount_path = "/etc/docker/certs.d/gitlab.example.com/ca.crt" - sub_path = "gitlab.example.com.crt" - ``` - -Replace `gitlab.example.com` with the actual domain of the registry. - -## Troubleshooting - -### Changing the default configuration has no effect - -A common issue is that the terms `Code Quality` (GitLab specific) and `Code Climate` -(Engine used by GitLab) are very similar. You must add a **`.codeclimate.yml`** file -to change the default configuration, **not** a `.codequality.yml` file. If you use -the wrong filename, the [default `.codeclimate.yml`](https://gitlab.com/gitlab-org/ci-cd/codequality/-/blob/master/codeclimate_defaults/.codeclimate.yml.template) -is still used. - -### No Code Quality report is displayed in a merge request - -This can be due to multiple reasons: - -- You just added the Code Quality job in your `.gitlab-ci.yml`. The report does not - have anything to compare to yet, so no information can be displayed. It only displays - after future merge requests have something to compare to. -- Your pipeline is not set to run the code quality job on your target branch. If there is no report generated from the target branch, your MR branch reports have nothing to compare to. In this situation you will see an error stating `Base pipeline codequality artifact not found`. -- If no [degradation or error is detected](https://docs.codeclimate.com/docs/maintainability#section-checks), - nothing is displayed. -- The [`artifacts:expire_in`](../../../ci/yaml/index.md#artifactsexpire_in) CI/CD - setting can cause the Code Quality artifacts to expire faster than desired. -- The widgets use the pipeline of the latest commit to the target branch. If commits are made to the default branch that do not run the code quality job, this may cause the merge request widget to have no base report for comparison. -- If you use the [`REPORT_STDOUT` environment variable](https://gitlab.com/gitlab-org/ci-cd/codequality#environment-variables), no report file is generated and nothing displays in the merge request. -- Large `gl-code-quality-report.json` files (esp. >10 MB) are [known to prevent the report from being displayed](https://gitlab.com/gitlab-org/gitlab/-/issues/2737). - As a work-around, try removing [properties](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types) - that are [ignored by GitLab](#implementing-a-custom-tool). You can: - - Configure the Code Quality tool to not output those types. - - Use `sed`, `awk` or similar commands in the `.gitlab-ci.yml` script to - edit the `gl-code-quality-report.json` before the job completes. - -### Only a single Code Quality report is displayed, but more are defined - -GitLab only uses the Code Quality artifact from the latest created job (with the largest job ID). -If multiple jobs in a pipeline generate a code quality artifact, those of earlier jobs are ignored. -To avoid confusion, configure only one job to generate a `gl-code-quality-report.json`. - -### RuboCop errors - -When using Code Quality jobs on a Ruby project, you can encounter problems running RuboCop. -For example, the following error can appear when using either a very recent or very old version -of Ruby: - -```plaintext -/usr/local/bundle/gems/rubocop-0.52.1/lib/rubocop/config.rb:510:in `check_target_ruby': -Unknown Ruby version 2.7 found in `.ruby-version`. (RuboCop::ValidationError) -Supported versions: 2.1, 2.2, 2.3, 2.4, 2.5 -``` - -This is caused by the default version of RuboCop used by the check engine not covering -support for the Ruby version in use. - -To use a custom version of RuboCop that -[supports the version of Ruby used by the project](https://docs.rubocop.org/rubocop/compatibility.html#support-matrix), -you can [override the configuration through a `.codeclimate.yml` file](https://docs.codeclimate.com/docs/rubocop#using-rubocops-newer-versions) -created in the project repository. - -For example, to specify using RuboCop release **0.67**: - -```yaml -version: "2" -plugins: - rubocop: - enabled: true - channel: rubocop-0-67 -``` - -### No Code Quality appears on merge requests when using custom tool - -If your merge requests do not show any code quality changes when using a custom tool, -ensure that the line property is an `integer`. - -### Code Quality CI job with Code Climate plugins enabled fails with error - -If you enabled any of the Code Climate plugins, and the Code Quality CI job fails with the error -below, it's likely the job takes longer than the default timeout of 900 seconds: - -```shell -error: (CC::CLI::Analyze::EngineFailure) engine pmd ran for 900 seconds and was killed -Could not analyze code quality for the repository at /code -``` - -To work around this problem, set `TIMEOUT_SECONDS` to a higher value in your `.gitlab.-ci.yml` file. - -For example: - -```yaml -variables: - TIMEOUT_SECONDS: 3600 -``` +<!-- This redirect file can be deleted after <2022-09-22>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/merge_requests/creating_merge_requests.md b/doc/user/project/merge_requests/creating_merge_requests.md index 6ee02238a22..f30b20e9d34 100644 --- a/doc/user/project/merge_requests/creating_merge_requests.md +++ b/doc/user/project/merge_requests/creating_merge_requests.md @@ -104,7 +104,7 @@ You can create a merge request from your fork to contribute back to the main pro After your work is merged, if you don't intend to make any other contributions to the upstream project, you can unlink your fork from its upstream project. Go to **Settings > Advanced Settings** and -[remove the forking relationship](../settings/index.md#removing-a-fork-relationship). +[remove the forking relationship](../settings/index.md#remove-a-fork-relationship). For more information, [see the forking workflow documentation](../repository/forking_workflow.md). diff --git a/doc/user/project/merge_requests/csv_export.md b/doc/user/project/merge_requests/csv_export.md index aaa9bec945f..2adcc4d4575 100644 --- a/doc/user/project/merge_requests/csv_export.md +++ b/doc/user/project/merge_requests/csv_export.md @@ -10,7 +10,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Exporting merge requests CSV enables you and your team to export all the data collected from merge requests into a comma-separated values (CSV) file, which stores tabular data in plain text. -To export merge requests to CSV, navigate to your **Merge requests** from the sidebar of a project and select **Export to CSV**. +To export merge requests to CSV, navigate to your **Merge requests** from the sidebar of a project and select **Export as CSV**. ## CSV Output diff --git a/doc/user/project/merge_requests/drafts.md b/doc/user/project/merge_requests/drafts.md index 13cc68f02dd..4bb6034c0bd 100644 --- a/doc/user/project/merge_requests/drafts.md +++ b/doc/user/project/merge_requests/drafts.md @@ -75,12 +75,10 @@ draft merge requests: ## Pipelines for drafts -When the [merged results pipelines](../../../ci/pipelines/merged_results_pipelines.md) -feature is enabled, draft merge requests run -[merge request pipelines](../../../ci/pipelines/merge_request_pipelines.md) only. +Draft merge requests run the same pipelines as merge request that are marked as ready. -To run merged results pipelines, you must -[mark the merge request as ready](#mark-merge-requests-as-ready). +In GitLab 15.0 and older, you must [mark the merge request as ready](#mark-merge-requests-as-ready) +if you want to run [merged results pipelines](../../../ci/pipelines/merged_results_pipelines.md). <!-- ## Troubleshooting diff --git a/doc/user/project/merge_requests/fail_fast_testing.md b/doc/user/project/merge_requests/fail_fast_testing.md index 355661516a7..c09a7c14c06 100644 --- a/doc/user/project/merge_requests/fail_fast_testing.md +++ b/doc/user/project/merge_requests/fail_fast_testing.md @@ -1,97 +1,11 @@ --- -stage: Verify -group: Pipeline Insights -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../../ci/testing/fail_fast_testing.md' +remove_date: '2022-08-31' --- -# Fail Fast Testing **(PREMIUM)** +This document was moved to [another location](../../../ci/testing/fail_fast_testing.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/198550) in GitLab 13.1. - -For applications that use RSpec for running tests, we've introduced the `Verify/Failfast` -[template to run subsets of your test suite](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Verify/FailFast.gitlab-ci.yml), -based on the changes in your merge request. - -The template uses the [test_file_finder (`tff`) gem](https://gitlab.com/gitlab-org/ci-cd/test_file_finder/) -that accepts a list of files as input, and returns a list of spec (test) files -that it believes to be relevant to the input files. - -`tff` is designed for Ruby on Rails projects, so the `Verify/FailFast` template is -configured to run when changes to Ruby files are detected. By default, it runs in -the [`.pre` stage](../../../ci/yaml/index.md#stage-pre) of a GitLab CI/CD pipeline, -before all other stages. - -## Example use case - -Fail fast testing is useful when adding new functionality to a project and adding -new automated tests. - -Your project could have hundreds of thousands of tests that take a long time to complete. -You may be confident that a new test will pass, but you have to wait for all the tests -to complete to verify it. This could take an hour or more, even when using parallelization. - -Fail fast testing gives you a faster feedback loop from the pipeline. It lets you -know quickly that the new tests are passing and the new functionality did not break -other tests. - -## Requirements - -This template requires: - -- A project built in Rails that uses RSpec for testing. -- CI/CD configured to: - - Use a Docker image with Ruby available. - - Use [Merge request pipelines](../../../ci/pipelines/merge_request_pipelines.md#prerequisites) -- [Merged results pipelines](../../../ci/pipelines/merged_results_pipelines.md#enable-merged-results-pipelines) - enabled in the project settings. -- A Docker image with Ruby available. The template uses `image: ruby:2.6` by default, but you [can override](../../../ci/yaml/includes.md#override-included-configuration-values) this. - -## Configuring Fast RSpec Failure - -We use the following plain RSpec configuration as a starting point. It installs all the -project gems and executes `rspec`, on merge request pipelines only. - -```yaml -rspec-complete: - stage: test - rules: - - if: $CI_PIPELINE_SOURCE == "merge_request_event" - script: - - bundle install - - bundle exec rspec -``` - -To run the most relevant specs first instead of the whole suite, [`include`](../../../ci/yaml/index.md#include) -the template by adding the following to your CI/CD configuration: - -```yaml -include: - - template: Verify/FailFast.gitlab-ci.yml -``` - -To customize the job, specific options may be set to override the template. For example, to override the default Docker image: - -```yaml -include: - - template: Verify/FailFast.gitlab-ci.yml - -rspec-rails-modified-path-specs: - image: custom-docker-image-with-ruby -``` - -### Example test loads - -For illustrative purposes, let's say our Rails app spec suite consists of 100 specs per model for ten models. - -If no Ruby files are changed: - -- `rspec-rails-modified-paths-specs` does not run any tests. -- `rspec-complete` runs the full suite of 1000 tests. - -If one Ruby model is changed, for example `app/models/example.rb`, then `rspec-rails-modified-paths-specs` -runs the 100 tests for `example.rb`: - -- If all of these 100 tests pass, then the full `rspec-complete` suite of 1000 tests is allowed to run. -- If any of these 100 tests fail, they fail quickly, and `rspec-complete` does not run any tests. - -The final case saves resources and time as the full 1000 test suite does not run. +<!-- This redirect file can be deleted after <2022-09-22>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/merge_requests/img/attention_request_list_v14_10.png b/doc/user/project/merge_requests/img/attention_request_list_v14_10.png Binary files differdeleted file mode 100644 index 00427a0aa40..00000000000 --- a/doc/user/project/merge_requests/img/attention_request_list_v14_10.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/attention_request_sidebar_v14_10.png b/doc/user/project/merge_requests/img/attention_request_sidebar_v14_10.png Binary files differdeleted file mode 100644 index 174cf01dbb0..00000000000 --- a/doc/user/project/merge_requests/img/attention_request_sidebar_v14_10.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/merge_method_ff_v15_0.png b/doc/user/project/merge_requests/img/merge_method_ff_v15_0.png Binary files differdeleted file mode 100644 index 323fd03ffa2..00000000000 --- a/doc/user/project/merge_requests/img/merge_method_ff_v15_0.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/merge_method_merge_commit_v15_0.png b/doc/user/project/merge_requests/img/merge_method_merge_commit_v15_0.png Binary files differdeleted file mode 100644 index b880c2c0e04..00000000000 --- a/doc/user/project/merge_requests/img/merge_method_merge_commit_v15_0.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/merge_method_merge_commit_with_semi_linear_history_v15_0.png b/doc/user/project/merge_requests/img/merge_method_merge_commit_with_semi_linear_history_v15_0.png Binary files differdeleted file mode 100644 index 9eab71e9d3c..00000000000 --- a/doc/user/project/merge_requests/img/merge_method_merge_commit_with_semi_linear_history_v15_0.png +++ /dev/null diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 30b69c2fff5..a7a669d3b75 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -251,60 +251,13 @@ This feature works only when a merge request is merged. Selecting **Remove sourc after merging does not retarget open merge requests. This improvement is [proposed as a follow-up](https://gitlab.com/gitlab-org/gitlab/-/issues/321559). -## Request attention to a merge request - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/343528) in GitLab 14.10 [with a flag](../../../administration/feature_flags.md) named `mr_attention_requests`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `mr_attention_requests`. -On GitLab.com, this feature is dependent on the enablement status of the feature flag. Refer to the [enablement issue](https://gitlab.com/gitlab-org/gitlab/-/issues/343528) for details. - -To tell a merge request's assignee or reviewer that their attention is -needed on a merge request, you can request their attention. If an assignee or a -reviewer has their attention requested on a merge request, the **Attention request** -icon (**{attention}**) is displayed as a solid icon (**{attention-solid}**) on -the merge request list page: - -![Attention request icon](img/attention_request_list_v14_10.png) - -To view a list of merge requests that need your attention: - -1. On the top bar, select **Merge requests** (**{merge-request}**). -1. Select **Attention requests**. - -To request attention from another user, use the `/attention @user` -[quick action](../quick_actions.md) or: - -1. Go to the merge request. -1. On the right sidebar, identify the user you want to request attention from. -1. Next to the user's name, select **Request attention** (**{attention}**), and the appearance - of the icon changes: - - ![Attention request toggle](img/attention_request_sidebar_v14_10.png) - -### Remove an attention request - -If your attention was requested as an assignee or reviewer, it's removed when you: - -- Manually remove the attention request by selecting **Remove attention request** (**{attention-solid}**). -- Approve the merge request. -- Add a new user as an assignee or reviewer. -- Request the attention of a different assignee or reviewer. -- Remove yourself (or are removed by someone else) as an assignee or reviewer. -- Merge or close the merge request. - -If you are both the assignee and a reviewer on a merge request, you receive -only one attention request, which is synced across both duties. If the -attention request is removed from you, either as an assignee or a reviewer, -it is removed from both your duties. - ## Merge request workflows For a software developer working in a team: 1. You checkout a new branch, and submit your changes through a merge request. 1. You gather feedback from your team. -1. You work on the implementation optimizing code with [Code Quality reports](code_quality.md). +1. You work on the implementation optimizing code with [Code Quality reports](../../../ci/testing/code_quality.md). 1. You verify your changes with [Unit test reports](../../../ci/testing/unit_test_reports.md) in GitLab CI/CD. 1. You avoid using dependencies whose license is not compatible with your project with [License Compliance reports](../../compliance/license_compliance/index.md). 1. You request the [approval](approvals/index.md) from your manager. diff --git a/doc/user/project/merge_requests/load_performance_testing.md b/doc/user/project/merge_requests/load_performance_testing.md index a5fff4a38be..04b62c5d8fe 100644 --- a/doc/user/project/merge_requests/load_performance_testing.md +++ b/doc/user/project/merge_requests/load_performance_testing.md @@ -1,201 +1,11 @@ --- -stage: Verify -group: Pipeline Insights -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../../ci/testing/load_performance_testing.md' +remove_date: '2022-08-31' --- -# Load Performance Testing **(PREMIUM)** +This document was moved to [another location](../../../ci/testing/load_performance_testing.md). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10683) in GitLab 13.2. - -With Load Performance Testing, you can test the impact of any pending code changes -to your application's backend in [GitLab CI/CD](../../../ci/index.md). - -GitLab uses [k6](https://k6.io/), a free and open source -tool, for measuring the system performance of applications under -load. - -Unlike [Browser Performance Testing](browser_performance_testing.md), which is -used to measure how web sites perform in client browsers, Load Performance Testing -can be used to perform various types of [load tests](https://k6.io/docs/#use-cases) -against application endpoints such as APIs, Web Controllers, and so on. -This can be used to test how the backend or the server performs at scale. - -For example, you can use Load Performance Testing to perform many concurrent -GET calls to a popular API endpoint in your application to see how it performs. - -## How Load Performance Testing works - -First, define a job in your `.gitlab-ci.yml` file that generates the -[Load Performance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsload_performance). -GitLab checks this report, compares key load performance metrics -between the source and target branches, and then shows the information in a merge request widget: - -![Load Performance Widget](img/load_performance_testing.png) - -Next, you need to configure the test environment and write the k6 test. - -The key performance metrics that the merge request widget shows after the test completes are: - -- Checks: The percentage pass rate of the [checks](https://k6.io/docs/using-k6/checks) configured in the k6 test. -- TTFB P90: The 90th percentile of how long it took to start receiving responses, aka the [Time to First Byte](https://en.wikipedia.org/wiki/Time_to_first_byte) (TTFB). -- TTFB P95: The 95th percentile for TTFB. -- RPS: The average requests per second (RPS) rate the test was able to achieve. - -NOTE: -If the Load Performance report has no data to compare, such as when you add the -Load Performance job in your `.gitlab-ci.yml` for the very first time, -the Load Performance report widget doesn't display. It must have run at least -once on the target branch (`main`, for example), before it displays in a -merge request targeting that branch. - -## Configure the Load Performance Testing job - -Configuring your Load Performance Testing job can be broken down into several distinct parts: - -- Determine the test parameters such as throughput, and so on. -- Set up the target test environment for load performance testing. -- Design and write the k6 test. - -### Determine the test parameters - -The first thing you need to do is determine the [type of load test](https://k6.io/docs/test-types/introduction) -you want to run, and how it will run (for example, the number of users, throughput, and so on). - -Refer to the [k6 docs](https://k6.io/docs/), especially the [k6 testing guides](https://k6.io/docs/testing-guides), -for guidance on the above and more. - -### Test Environment setup - -A large part of the effort around load performance testing is to prepare the target test environment -for high loads. You should ensure it's able to handle the -[throughput](https://k6.io/blog/monthly-visits-concurrent-users) it will be tested with. - -It's also typically required to have representative test data in the target environment -for the load performance test to use. - -We strongly recommend [not running these tests against a production environment](https://k6.io/our-beliefs#load-test-in-a-pre-production-environment). - -### Write the load performance test - -After the environment is prepared, you can write the k6 test itself. k6 is a flexible -tool and can be used to run [many kinds of performance tests](https://k6.io/docs/test-types/introduction). -Refer to the [k6 documentation](https://k6.io/docs/) for detailed information on how to write tests. - -### Configure the test in GitLab CI/CD - -When your k6 test is ready, the next step is to configure the load performance -testing job in GitLab CI/CD. The easiest way to do this is to use the -[`Verify/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Verify/Load-Performance-Testing.gitlab-ci.yml) -template that is included with GitLab. - -NOTE: -For large scale k6 tests you need to ensure the GitLab Runner instance performing the actual -test is able to handle running the test. Refer to [k6's guidance](https://k6.io/docs/testing-guides/running-large-tests#hardware-considerations) -for spec details. The [default shared GitLab.com runners](../../../ci/runners/saas/linux_saas_runner.md) -likely have insufficient specs to handle most large k6 tests. - -This template runs the -[k6 Docker container](https://hub.docker.com/r/loadimpact/k6/) in the job and provides several ways to customize the -job. - -An example configuration workflow: - -1. Set up GitLab Runner to run Docker containers, like the - [Docker-in-Docker workflow](../../../ci/docker/using_docker_build.md#use-docker-in-docker). -1. Configure the default Load Performance Testing CI/CD job in your `.gitlab-ci.yml` file. - You need to include the template and configure it with CI/CD variables: - - ```yaml - include: - template: Verify/Load-Performance-Testing.gitlab-ci.yml - - load_performance: - variables: - K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT> - ``` - -The above example creates a `load_performance` job in your CI/CD pipeline that runs -the k6 test. - -NOTE: -For Kubernetes setups a different template should be used: [`Jobs/Load-Performance-Testing.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Load-Performance-Testing.gitlab-ci.yml). - -k6 has [various options](https://k6.io/docs/using-k6/options) to configure how it will run tests, such as what throughput (RPS) to run with, -how long the test should run, and so on. Almost all options can be configured in the test itself, but as -you can also pass command line options via the `K6_OPTIONS` variable. - -For example, you can override the duration of the test with a CLI option: - -```yaml - include: - template: Verify/Load-Performance-Testing.gitlab-ci.yml - - load_performance: - variables: - K6_TEST_FILE: <PATH TO K6 TEST FILE IN PROJECT> - K6_OPTIONS: '--duration 30s' -``` - -GitLab only displays the key performance metrics in the MR widget if k6's results are saved -via [summary export](https://k6.io/docs/results-visualization/json#summary-export) -as a [Load Performance report artifact](../../../ci/yaml/artifacts_reports.md#artifactsreportsload_performance). -The latest Load Performance artifact available is always used, using the -summary values from the test. - -If [GitLab Pages](../pages/index.md) is enabled, you can view the report directly in your browser. - -### Load Performance testing in Review Apps - -The CI/CD YAML configuration example above works for testing against static environments, -but it can be extended to work with [review apps](../../../ci/review_apps) or -[dynamic environments](../../../ci/environments) with a few extra steps. - -The best approach is to capture the dynamic URL in a [`.env` file](https://docs.docker.com/compose/env-file/) -as a job artifact to be shared, then use a custom CI/CD variable we've provided named `K6_DOCKER_OPTIONS` -to configure the k6 Docker container to use the file. With this, k6 can then use any -environment variables from the `.env` file in scripts using standard JavaScript, -such as: ``http.get(`${__ENV.ENVIRONMENT_URL}`)``. - -For example: - -1. In the `review` job: - 1. Capture the dynamic URL and save it into a `.env` file, for example, `echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env`. - 1. Set the `.env` file to be a [job artifact](../../../ci/pipelines/job_artifacts.md#job-artifacts). -1. In the `load_performance` job: - 1. Set it to depend on the review job, so it inherits the environment file. - 1. Set the `K6_DOCKER_OPTIONS` variable with the [Docker CLI option for environment files](https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file), for example `--env-file review.env`. -1. Configure the k6 test script to use the environment variable in it's steps. - -Your `.gitlab-ci.yml` file might be similar to: - -```yaml -stages: - - deploy - - performance - -include: - template: Verify/Load-Performance-Testing.gitlab-ci.yml - -review: - stage: deploy - environment: - name: review/$CI_COMMIT_REF_SLUG - url: http://$CI_ENVIRONMENT_SLUG.example.com - script: - - run_deploy_script - - echo "ENVIRONMENT_URL=$CI_ENVIRONMENT_URL" >> review.env - artifacts: - paths: - - review.env - rules: - - if: $CI_COMMIT_BRANCH # Modify to match your pipeline rules, or use `only/except` if needed. - -load_performance: - dependencies: - - review - variables: - K6_DOCKER_OPTIONS: '--env-file review.env' - rules: - - if: $CI_COMMIT_BRANCH # Modify to match your pipeline rules, or use `only/except` if needed. -``` +<!-- This redirect file can be deleted after <2022-09-22>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/merge_requests/methods/index.md b/doc/user/project/merge_requests/methods/index.md index d3221162cfd..63b464e5ff4 100644 --- a/doc/user/project/merge_requests/methods/index.md +++ b/doc/user/project/merge_requests/methods/index.md @@ -23,7 +23,26 @@ merge requests are merged into an existing branch. This setting is the default. It always creates a separate merge commit, even when using [squash](../squash_and_merge.md). An example commit graph generated using this merge method: -![Commit graph for merge commits](../img/merge_method_merge_commit_v15_0.png) +```mermaid +gitGraph + commit id: "Init" + branch mr-branch-1 + commit + checkout main + commit + branch mr-branch-2 + commit + checkout mr-branch-1 + commit + checkout main + branch squash-mr + commit id: "Squashed commits" + checkout main + merge squash-mr + merge mr-branch-1 + commit + merge mr-branch-2 +``` - For regular merges, it is equivalent to the command `git merge --no-ff <source-branch>`. - For squash merges, it squashes all commits in the source branch before merging it normally. It performs actions similar to: @@ -42,7 +61,25 @@ A merge commit is created for every merge, but the branch is only merged if a fast-forward merge is possible. This ensures that if the merge request build succeeded, the target branch build also succeeds after the merge. An example commit graph generated using this merge method: -![Commit graph for merge commit with semi-linear history](../img/merge_method_merge_commit_with_semi_linear_history_v15_0.png) +```mermaid +gitGraph + commit id: "Init" + branch mr-branch-1 + commit + commit + checkout main + merge mr-branch-1 + branch mr-branch-2 + commit + commit + checkout main + merge mr-branch-2 + commit + branch squash-mr + commit id: "Squashed commits" + checkout main + merge squash-mr +``` When you visit the merge request page with `Merge commit with semi-linear history` method selected, you can accept it **only if a fast-forward merge is possible**. @@ -63,7 +100,14 @@ fast-forward merge requests, you can retain a linear Git history and a way to accept merge requests without creating merge commits. An example commit graph generated using this merge method: -![Commit graph for fast-forward merge](../img/merge_method_ff_v15_0.png) +```mermaid +gitGraph + commit id: "Init" + commit id: "Merge mr-branch-1" + commit id: "Merge mr-branch-2" + commit id: "Commit on main" + commit id: "Merge squash-mr" +``` This method is equivalent to `git merge --ff <source-branch>` for regular merges, and to `git merge -squash <source-branch>` for squash merges. diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index 8f77ce90436..a8f43dd9c02 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -112,13 +112,7 @@ This example shows reviewers and approval rules in a merge request sidebar: ### Request a new review -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293933) in GitLab 13.9. -> - [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/357271) in GitLab 14.10. - -WARNING: -This feature is in its end-of-life process. It is [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/357271) -in GitLab 14.10, and is planned for [removal](https://gitlab.com/gitlab-org/gitlab/-/issues/357271) in GitLab 15.0. -Use [attention requests](../index.md#request-attention-to-a-merge-request) instead. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293933) in GitLab 13.9. After a reviewer completes their [merge request reviews](../../../discussions/index.md), the author of the merge request can request a new review from the reviewer: diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index 7360b57103b..2ff65571c8b 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -77,7 +77,7 @@ in four backticks instead of three: ## Configure the commit message for applied suggestions -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13086) in GitLab 12.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13086) in GitLab 12.7. GitLab uses a default commit message when applying suggestions: `Apply %{suggestions_count} suggestion(s) to %{files_count} file(s)` diff --git a/doc/user/project/merge_requests/status_checks.md b/doc/user/project/merge_requests/status_checks.md index 76a67487881..423179325d3 100644 --- a/doc/user/project/merge_requests/status_checks.md +++ b/doc/user/project/merge_requests/status_checks.md @@ -138,7 +138,7 @@ the status check and it **will not** be recoverable. ## Status checks widget -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327634) in GitLab 14.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327634) in GitLab 14.1. The status checks widget displays in merge requests and shows the status of external status checks: diff --git a/doc/user/project/merge_requests/test_coverage_visualization.md b/doc/user/project/merge_requests/test_coverage_visualization.md index fcbd732f8ee..53d45e6940d 100644 --- a/doc/user/project/merge_requests/test_coverage_visualization.md +++ b/doc/user/project/merge_requests/test_coverage_visualization.md @@ -1,441 +1,11 @@ --- -stage: Verify -group: Pipeline Insights -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../../ci/testing/test_coverage_visualization.md' +remove_date: '2022-08-31' --- -# Test coverage visualization **(FREE)** +This document was moved to [another location](../../../ci/testing/test_coverage_visualization.md). -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/249811) in GitLab 13.5. - -With the help of [GitLab CI/CD](../../../ci/index.md), you can collect the test -coverage information of your favorite testing or coverage-analysis tool, and visualize -this information inside the file diff view of your merge requests (MRs). This will allow you -to see which lines are covered by tests, and which lines still require coverage, before the -MR is merged. - -![Test Coverage Visualization Diff View](img/test_coverage_visualization_v12_9.png) - -## How test coverage visualization works - -Collecting the coverage information is done via GitLab CI/CD's -[artifacts reports feature](../../../ci/yaml/index.md#artifactsreports). -You can specify one or more coverage reports to collect, including wildcard paths. -GitLab then takes the coverage information in all the files and combines it -together. Coverage files are parsed in a background job so there can be a delay -between pipeline completion and the visualization loading on the page. - -For the coverage analysis to work, you have to provide a properly formatted -[Cobertura XML](https://cobertura.github.io/cobertura/) report to -[`artifacts:reports:coverage_report`](../../../ci/yaml/artifacts_reports.md#artifactsreportscoverage_report). -This format was originally developed for Java, but most coverage analysis frameworks -for other languages have plugins to add support for it, like: - -- [simplecov-cobertura](https://rubygems.org/gems/simplecov-cobertura) (Ruby) -- [gocover-cobertura](https://github.com/boumenot/gocover-cobertura) (Golang) - -Other coverage analysis frameworks support the format out of the box, for example: - -- [Istanbul](https://istanbul.js.org/docs/advanced/alternative-reporters/#cobertura) (JavaScript) -- [Coverage.py](https://coverage.readthedocs.io/en/coverage-5.0.4/cmd.html#xml-reporting) (Python) -- [PHPUnit](https://github.com/sebastianbergmann/phpunit-documentation-english/blob/master/src/textui.rst#command-line-options) (PHP) - -Once configured, if you create a merge request that triggers a pipeline which collects -coverage reports, the coverage is shown in the diff view. This includes reports -from any job in any stage in the pipeline. The coverage displays for each line: - -- `covered` (green): lines which have been checked at least once by tests -- `no test coverage` (orange): lines which are loaded but never executed -- no coverage information: lines which are non-instrumented or not loaded - -Hovering over the coverage bar provides further information, such as the number -of times the line was checked by tests. - -Uploading a test coverage report does not enable: - -- [Test coverage results in merge requests](../../../ci/pipelines/settings.md#merge-request-test-coverage-results). -- [Code coverage history](../../../ci/pipelines/settings.md#view-code-coverage-history). - -You must configure these separately. - -### Limits - -A limit of 100 `<source>` nodes for Cobertura format XML files applies. If your Cobertura report exceeds -100 nodes, there can be mismatches or no matches in the merge request diff view. - -A single Cobertura XML file can be no more than 10MiB. For large projects, split the Cobertura XML into -smaller files. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328772) for more details. -When submitting many files, it can take a few minutes for coverage to show on a merge request. - -The visualization only displays after the pipeline is complete. If the pipeline has -a [blocking manual job](../../../ci/jobs/job_control.md#types-of-manual-jobs), the -pipeline waits for the manual job before continuing and is not considered complete. -The visualization cannot be displayed if the blocking manual job did not run. - -### Artifact expiration - -By default, the [pipeline artifact](../../../ci/pipelines/pipeline_artifacts.md#storage) used -to draw the visualization on the merge request expires **one week** after creation. - -### Coverage report from child pipeline - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/363301) in GitLab 15.1 [with a flag](../../../administration/feature_flags.md). Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `ci_child_pipeline_coverage_reports`. -On GitLab.com, this feature is not available. -The feature is not ready for production use. - -If the test coverage is created in jobs that are in a child pipeline, the parent pipeline must use -`strategy: depend`. - -```yaml -child_test_pipeline: - trigger: - include: - - local: path/to/child_pipeline.yml - - template: Security/SAST.gitlab-ci.yml - strategy: depend -``` - -### Automatic class path correction - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217664) in GitLab 13.8. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284822) in GitLab 13.9. - -The coverage report properly matches changed files only if the `filename` of a `class` element -contains the full path relative to the project root. However, in some coverage analysis frameworks, -the generated Cobertura XML has the `filename` path relative to the class package directory instead. - -To make an intelligent guess on the project root relative `class` path, the Cobertura XML parser -attempts to build the full path by: - -- Extracting a portion of the `source` paths from the `sources` element and combining them with the - class `filename` path. -- Checking if the candidate path exists in the project. -- Using the first candidate that matches as the class full path. - -#### Path correction example - -As an example, a project with: - -- A full path of `test-org/test-project`. -- The following files relative to the project root: - - ```shell - Auth/User.cs - Lib/Utils/User.cs - src/main/java - ``` - -In the: - -- Cobertura XML, the `filename` attribute in the `class` element assumes the value is a relative - path to the project's root: - - ```xml - <class name="packet.name" filename="src/main/java" line-rate="0.0" branch-rate="0.0" complexity="5"> - ``` - -- `sources` from Cobertura XML, the following paths in the format - `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`: - - ```xml - <sources> - <source>/builds/test-org/test-project/Auth</source> - <source>/builds/test-org/test-project/Lib/Utils</source> - </sources> - ``` - -The parser: - -- Extracts `Auth` and `Lib/Utils` from the `sources` and uses these to determine the `class` path - relative to the project root. -- Combines these extracted `sources` and the class filename. For example, if there is a `class` - element with the `filename` value of `User.cs`, the parser takes the first candidate path that - matches, which is `Auth/User.cs`. -- For each `class` element, attempts to look for a match for each extracted `source` path up to - 100 iterations. If it reaches this limit without finding a matching path in the file tree, the - class is not included in the final coverage report. - -NOTE: -Automatic class path correction only works on `source` paths in the format `<CI_BUILDS_DIR>/<PROJECT_FULL_PATH>/...`. -The `source` is ignored if the path does not follow this pattern. The parser assumes that the -`filename` of a `class` element contains the full path relative to the project root. - -## Example test coverage configurations - -This section provides test coverage configuration examples for different programming languages. You can also see a working example in -the [`coverage-report`](https://gitlab.com/gitlab-org/ci-sample-projects/coverage-report/) demonstration project. - -### JavaScript example - -The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example uses [Mocha](https://mochajs.org/) -JavaScript testing and [nyc](https://github.com/istanbuljs/nyc) coverage-tooling to -generate the coverage artifact: - -```yaml -test: - script: - - npm install - - npx nyc --reporter cobertura mocha - artifacts: - reports: - coverage_report: - coverage_format: cobertura - path: coverage/cobertura-coverage.xml -``` - -### Java and Kotlin examples - -#### Maven example - -The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Java or Kotlin uses [Maven](https://maven.apache.org/) -to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to -generate the coverage artifact. -You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image. - -GitLab expects the artifact in the Cobertura format, so you have to execute a few -scripts before uploading it. The `test-jdk11` job tests the code and generates an -XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report: - -```yaml -test-jdk11: - stage: test - image: maven:3.6.3-jdk-11 - script: - - mvn $MAVEN_CLI_OPTS clean org.jacoco:jacoco-maven-plugin:prepare-agent test jacoco:report - artifacts: - paths: - - target/site/jacoco/jacoco.xml - -coverage-jdk11: - # Must be in a stage later than test-jdk11's stage. - # The `visualize` stage does not exist by default. - # Please define it first, or choose an existing stage like `deploy`. - stage: visualize - image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7 - script: - # convert report from jacoco to cobertura, using relative project path - - python /opt/cover2cover.py target/site/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > target/site/cobertura.xml - needs: ["test-jdk11"] - artifacts: - reports: - coverage_report: - coverage_format: cobertura - path: target/site/cobertura.xml -``` - -#### Gradle example - -The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Java or Kotlin uses [Gradle](https://gradle.org/) -to build the project and [JaCoCo](https://www.eclemma.org/jacoco/) coverage-tooling to -generate the coverage artifact. -You can check the [Docker image configuration and scripts](https://gitlab.com/haynes/jacoco2cobertura) if you want to build your own image. - -GitLab expects the artifact in the Cobertura format, so you have to execute a few -scripts before uploading it. The `test-jdk11` job tests the code and generates an -XML artifact. The `coverage-jdk-11` job converts the artifact into a Cobertura report: - -```yaml -test-jdk11: - stage: test - image: gradle:6.6.1-jdk11 - script: - - 'gradle test jacocoTestReport' # jacoco must be configured to create an xml report - artifacts: - paths: - - build/jacoco/jacoco.xml - -coverage-jdk11: - # Must be in a stage later than test-jdk11's stage. - # The `visualize` stage does not exist by default. - # Please define it first, or chose an existing stage like `deploy`. - stage: visualize - image: registry.gitlab.com/haynes/jacoco2cobertura:1.0.7 - script: - # convert report from jacoco to cobertura, using relative project path - - python /opt/cover2cover.py build/jacoco/jacoco.xml $CI_PROJECT_DIR/src/main/java/ > build/cobertura.xml - needs: ["test-jdk11"] - artifacts: - reports: - coverage_report: - coverage_format: cobertura - path: build/cobertura.xml -``` - -### Python example - -The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Python uses [pytest-cov](https://pytest-cov.readthedocs.io/) to collect test coverage data and [coverage.py](https://coverage.readthedocs.io/) to convert the report to use full relative paths. -The information isn't displayed without the conversion. - -This example assumes that the code for your package is in `src/` and your tests are in `tests.py`: - -```yaml -run tests: - stage: test - image: python:3 - script: - - pip install pytest pytest-cov - - coverage run -m pytest - - coverage report - - coverage xml - coverage: '/(?i)total.*? (100(?:\.0+)?\%|[1-9]?\d(?:\.\d+)?\%)$/' - artifacts: - reports: - coverage_report: - coverage_format: cobertura - path: coverage.xml -``` - -### PHP example - -The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for PHP uses [PHPUnit](https://phpunit.readthedocs.io/) -to collect test coverage data and generate the report. - -With a minimal [`phpunit.xml`](https://phpunit.readthedocs.io/en/9.5/configuration.html) file (you may reference -[this example repository](https://gitlab.com/yookoala/code-coverage-visualization-with-php/)), you can run the test and -generate the `coverage.xml`: - -```yaml -run tests: - stage: test - image: php:latest - variables: - XDEBUG_MODE: coverage - before_script: - - apt-get update && apt-get -yq install git unzip zip libzip-dev zlib1g-dev - - docker-php-ext-install zip - - pecl install xdebug && docker-php-ext-enable xdebug - - php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');" - - php composer-setup.php --install-dir=/usr/local/bin --filename=composer - - composer install - - composer require --dev phpunit/phpunit phpunit/php-code-coverage - script: - - php ./vendor/bin/phpunit --coverage-text --coverage-cobertura=coverage.cobertura.xml - artifacts: - reports: - coverage_report: - coverage_format: cobertura - path: coverage.cobertura.xml -``` - -[Codeception](https://codeception.com/), through PHPUnit, also supports generating Cobertura report with -[`run`](https://codeception.com/docs/reference/Commands#run). The path for the generated file -depends on the `--coverage-cobertura` option and [`paths`](https://codeception.com/docs/reference/Configuration#paths) -configuration for the [unit test suite](https://codeception.com/docs/05-UnitTests). Configure `.gitlab-ci.yml` -to find Cobertura in the appropriate path. - -### C/C++ example - -The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for C/C++ with -`gcc` or `g++` as the compiler uses [`gcovr`](https://gcovr.com/en/stable/) to generate the coverage -output file in Cobertura XML format. - -This example assumes: - -- That the `Makefile` is created by `cmake` in the `build` directory, - within another job in a previous stage. - (If you use `automake` to generate the `Makefile`, - then you need to call `make check` instead of `make test`.) -- `cmake` (or `automake`) has set the compiler option `--coverage`. - -```yaml -run tests: - stage: test - script: - - cd build - - make test - - gcovr --xml-pretty --exclude-unreachable-branches --print-summary -o coverage.xml --root ${CI_PROJECT_DIR} - coverage: /^\s*lines:\s*\d+.\d+\%/ - artifacts: - name: ${CI_JOB_NAME}-${CI_COMMIT_REF_NAME}-${CI_COMMIT_SHA} - expire_in: 2 days - reports: - coverage_report: - coverage_format: cobertura - path: build/coverage.xml -``` - -### Go example - -The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Go uses: - -- [`go test`](https://go.dev/doc/tutorial/add-a-test) to run tests. -- [`gocover-cobertura`](https://github.com/boumenot/gocover-cobertura) to convert Go's coverage profile into the Cobertura XML format. - -This example assumes that [Go modules](https://go.dev/ref/mod) -are being used. Please note that the `-covermode count` option does not work with the `-race` flag. -If you want to generate code coverage while also using the `-race` flag, you must switch to -`-covermode atomic` which is slower than `-covermode count`. See [this blog post](https://go.dev/blog/cover) -for more details. - -```yaml -run tests: - stage: test - image: golang:1.17 - script: - - go install - - go test ./... -coverprofile=coverage.txt -covermode count - - go get github.com/boumenot/gocover-cobertura - - go run github.com/boumenot/gocover-cobertura < coverage.txt > coverage.xml - artifacts: - reports: - coverage_report: - coverage_format: cobertura - path: coverage.xml -``` - -### Ruby example - -The following [`.gitlab-ci.yml`](../../../ci/yaml/index.md) example for Ruby uses - -- [`rspec`](https://rspec.info/) to run tests. -- [`simplecov`](https://github.com/simplecov-ruby/simplecov) and [`simplecov-cobertura`](https://github.com/dashingrocket/simplecov-cobertura) - to record the coverage profile and create a report in the Cobertura XML format. - -This example assumes: - -- That [`bundler`](https://bundler.io/) is being used for dependency management. - The `rspec`, `simplecov` and `simplecov-cobertura` gems have been added to your `Gemfile`. -- The `CoberturaFormatter` has been added to your `SimpleCov.formatters` - configuration within the `spec_helper.rb` file. - -```yaml -run tests: - stage: test - image: ruby:3.1 - script: - - bundle install - - bundle exec rspec - artifacts: - reports: - coverage_report: - coverage_format: cobertura - path: coverage/coverage.xml -``` - -## Troubleshooting - -### Test coverage visualization not displayed - -If the test coverage visualization is not displayed in the diff view, you can check -the coverage report itself and verify that: - -- The file you are viewing in the diff view is mentioned in the coverage report. -- The `source` and `filename` nodes in the report follows the [expected structure](#automatic-class-path-correction) - to match the files in your repository. - -Report artifacts are not downloadable by default. If you want the report to be downloadable -from the job details page, add your coverage report to the artifact `paths`: - -```yaml -artifacts: - paths: - - coverage/cobertura-coverage.xml - reports: - coverage_report: - coverage_format: cobertura - path: coverage/cobertura-coverage.xml -``` +<!-- This redirect file can be deleted after <2022-09-22>. --> +<!-- Redirects that point to other docs in the same project expire in three months. --> +<!-- Redirects that point to docs in a different project or site (for example, link is not relative and starts with `https:`) expire in one year. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/redirects.html --> diff --git a/doc/user/project/milestones/burndown_and_burnup_charts.md b/doc/user/project/milestones/burndown_and_burnup_charts.md index d6fcd9fbb16..0f36747a547 100644 --- a/doc/user/project/milestones/burndown_and_burnup_charts.md +++ b/doc/user/project/milestones/burndown_and_burnup_charts.md @@ -9,7 +9,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w [Burndown](#burndown-charts) and [burnup](#burnup-charts) charts show the progress of completing a milestone. -![burndown and burnup chart](img/burndown_and_burnup_charts_v15_1.png) +![burndown and burnup chart](img/burndown_and_burnup_charts_v15_3.png) ## Burndown charts @@ -19,7 +19,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w Burndown charts show the number of issues over the course of a milestone. -![burndown chart](img/burndown_chart_v15_1.png) +![burndown chart](img/burndown_chart_v15_3.png) At a glance, you see the current state for the completion a given milestone. Without them, you would have to organize the data from the milestone and plot it @@ -66,7 +66,7 @@ A burndown chart is available for every project or group milestone that has been date** and a **due date**. NOTE: -You're able to [promote project](index.md#promoting-project-milestones-to-group-milestones) to group milestones and still see the **burndown chart** for them, respecting license limitations. +You're able to [promote project](index.md#promote-a-project-milestone-to-a-group-milestone) to group milestones and still see the **burndown chart** for them, respecting license limitations. The chart indicates the project's progress throughout that milestone (for issues assigned to it). @@ -106,7 +106,7 @@ Reopened issues are considered as having been opened on the day after they were Burnup charts show the assigned and completed work for a milestone. -![burnup chart](img/burnup_chart_v15_1.png) +![burnup chart](img/burnup_chart_v15_3.png) To view a project's burnup chart: diff --git a/doc/user/project/milestones/img/burndown_and_burnup_charts_v15_1.png b/doc/user/project/milestones/img/burndown_and_burnup_charts_v15_1.png Binary files differdeleted file mode 100644 index 58c0ddf892f..00000000000 --- a/doc/user/project/milestones/img/burndown_and_burnup_charts_v15_1.png +++ /dev/null diff --git a/doc/user/project/milestones/img/burndown_and_burnup_charts_v15_3.png b/doc/user/project/milestones/img/burndown_and_burnup_charts_v15_3.png Binary files differnew file mode 100644 index 00000000000..1420123500c --- /dev/null +++ b/doc/user/project/milestones/img/burndown_and_burnup_charts_v15_3.png diff --git a/doc/user/project/milestones/img/burndown_chart_v15_1.png b/doc/user/project/milestones/img/burndown_chart_v15_1.png Binary files differdeleted file mode 100644 index 2953380292d..00000000000 --- a/doc/user/project/milestones/img/burndown_chart_v15_1.png +++ /dev/null diff --git a/doc/user/project/milestones/img/burndown_chart_v15_3.png b/doc/user/project/milestones/img/burndown_chart_v15_3.png Binary files differnew file mode 100644 index 00000000000..9e1c7ccd4dd --- /dev/null +++ b/doc/user/project/milestones/img/burndown_chart_v15_3.png diff --git a/doc/user/project/milestones/img/burnup_chart_v15_1.png b/doc/user/project/milestones/img/burnup_chart_v15_1.png Binary files differdeleted file mode 100644 index e89b76344ed..00000000000 --- a/doc/user/project/milestones/img/burnup_chart_v15_1.png +++ /dev/null diff --git a/doc/user/project/milestones/img/burnup_chart_v15_3.png b/doc/user/project/milestones/img/burnup_chart_v15_3.png Binary files differnew file mode 100644 index 00000000000..2e85c0abe87 --- /dev/null +++ b/doc/user/project/milestones/img/burnup_chart_v15_3.png diff --git a/doc/user/project/milestones/img/milestones_promote_milestone.png b/doc/user/project/milestones/img/milestones_promote_milestone.png Binary files differdeleted file mode 100644 index 2ef85c5951d..00000000000 --- a/doc/user/project/milestones/img/milestones_promote_milestone.png +++ /dev/null diff --git a/doc/user/project/milestones/index.md b/doc/user/project/milestones/index.md index b0f3179961d..ba48876d4fd 100644 --- a/doc/user/project/milestones/index.md +++ b/doc/user/project/milestones/index.md @@ -23,87 +23,127 @@ Additionally, you can integrate milestones with the [Releases feature](../releas ## Project milestones and group milestones -You can assign **project milestones** to issues or merge requests in that project only. -To view the project milestone list, in a project, go to **{issues}** **Issues > Milestones**. +A milestone can belong to [project](../index.md) or [group](../../group/index.md). +You can assign **project milestones** to issues or merge requests in that project only. You can assign **group milestones** to any issue or merge request of any project in that group. -To view the group milestone list, in a group, go to **{issues}** **Issues > Milestones**. - -You can also view all milestones you have access to in the dashboard milestones list. -To view both project milestones and group milestones you have access to, select **Menu > Milestones** -on the top bar. For information about project and group milestones API, see: - [Project Milestones API](../../../api/milestones.md) - [Group Milestones API](../../../api/group_milestones.md) -NOTE: -If you're in a group and select **Issues > Milestones**, GitLab displays group milestones -and the milestones of projects in this group. -If you're in a project and select **Issues > Milestones**, GitLab displays only this project's milestones. +### View project or group milestones + +To view the milestone list: + +1. On the top bar, select **Menu > Projects** and find your project or + **Menu > Groups** and find your group. +1. Select **Issues > Milestones**. + +In a project, GitLab displays milestones that belong to the project. +In a group, GitLab displays milestones that belong to the group and all projects in the group. + +### View all milestones -## Creating milestones +You can view all the milestones you have access to in the entire GitLab namespace. +You might not see some milestones because they're in projects or groups you're not a member of. + +To do so, on the top bar select **Menu > Milestones**. + +## Create a milestone > [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/343889) the minimum user role from Developer to Reporter in GitLab 15.0. -Milestones can be created either at project or group level. +You can create a milestone either in a project or a group. Prerequisites: -- You must have at least the Reporter role for a group. +- You must have at least the Reporter role for the project or group the milestone belongs to. To create a milestone: 1. On the top bar, select **Menu > Projects** and find your project or **Menu > Groups** and find your group. 1. On the left sidebar, select **Issues > Milestones**. 1. Select **New milestone**. -1. Enter the title, an optional description, an optional start date, and an optional due date. +1. Enter the title. +1. Optional. Enter description, start date, and due date. 1. Select **New milestone**. ![New milestone](img/milestones_new_project_milestone.png) -## Editing milestones +## Edit a milestone > [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/343889) the minimum user role from Developer to Reporter in GitLab 15.0. -Users with at least the Reporter role can edit milestones. - Prerequisites: -- You must have at least the Reporter role for a group. +- You must have at least the Reporter role for the project or group the milestone belongs to. To edit a milestone: -1. In a project or group, go to **{issues}** **Issues > Milestones**. +1. On the top bar, select **Menu > Projects** and find your project or **Menu > Groups** and find your group. 1. Select a milestone's title. 1. Select **Edit**. +1. Edit the title, start date, due date, or description. +1. Select **Save changes**. + +## Delete a milestone + +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/343889) the minimum user role from Developer to Reporter in GitLab 15.0. -You can delete a milestone by selecting the **Delete** button. +Prerequisites: + +- You must have at least the Reporter role for the project or group the milestone belongs to. + +To edit a milestone: + +1. On the top bar, select **Menu > Projects** and find your project or **Menu > Groups** and find your group. +1. Select a milestone's title. +1. Select **Delete**. +1. Select **Delete milestone**. -### Promoting project milestones to group milestones +## Promote a project milestone to a group milestone If you are expanding the number of projects in a group, you might want to share the same milestones -among this group's projects. You can also promote project milestones to group milestones in order to +among this group's projects. You can also promote project milestones to group milestones to make them available to other projects in the same group. -From the project milestone list page, you can promote a project milestone to a group milestone. -This merges all project milestones across all projects in this group with the same name into a single -group milestones. All issues and merge requests that were previously assigned to one of these project -milestones is assigned the new group milestones. This action cannot be reversed and the changes are -permanent. +Promoting a milestone merges all project milestones across all projects in this group with the same +name into a single group milestone. +All issues and merge requests that were previously assigned to one of these project +milestones become assigned to the new group milestone. WARNING: -From GitLab 12.4 and earlier, some information is lost when you promote a project milestone to a -group milestone. Not all features on the project milestone view are available on the group milestone -view. If you promote a project milestone to a group milestone, you lose these features. Visit -[Milestone view](#milestone-view) to learn which features are missing from the group milestone view. +This action cannot be reversed and the changes are permanent. + +Prerequisites: + +- You must have at least the Reporter role for the group. + +To promote a project milestone: + +1. On the top bar, select **Menu > Projects** and find your project. +1. Either: + - Select **Promote to Group Milestone** (**{level-up}**). + - Select the milestone title, and then select **Promote**. +1. Select **Promote Milestone**. + +## Assign a milestone to an issue or merge request + +Every issue and merge request can be assigned one milestone. +The milestones are visible on every issue and merge request page, on the right sidebar. +They are also visible in the issue board. -![Promote milestone](img/milestones_promote_milestone.png) +To assign or unassign a milestone: -## Assigning milestones from the sidebar +1. View an issue or a merge request. +1. On the right sidebar, next to **Milestones**, select **Edit**. +1. In the **Assign milestone** list, search for a milestone by typing its name. + You can select from both project and group milestones. +1. Select the milestone you want to assign. -Every issue and merge request can be assigned a milestone. The milestones are visible on every issue and merge request page, in the sidebar. They are also visible in the issue board. From the sidebar, you can assign or unassign a milestones to the object. You can also perform this as a [quick action](../quick_actions.md) in a comment. [As mentioned](#project-milestones-and-group-milestones), for a given issue or merge request, both project milestones and group milestones can be selected and assigned to the object. +You can also use the `/assign` [quick action](../quick_actions.md) in a comment. ## Filtering issues and merge requests by milestone @@ -156,7 +196,7 @@ There are also tabs below these that show the following: The milestone view contains a [burndown and burnup chart](burndown_and_burnup_charts.md), showing the progress of completing a milestone. -![burndown chart](img/burndown_and_burnup_charts_v15_1.png) +![burndown chart](img/burndown_and_burnup_charts_v15_3.png) ### Milestone sidebar diff --git a/doc/user/project/pages/redirects.md b/doc/user/project/pages/redirects.md index 791b6a1550a..5d03db4bf00 100644 --- a/doc/user/project/pages/redirects.md +++ b/doc/user/project/pages/redirects.md @@ -45,8 +45,9 @@ Note that: - All paths must start with a forward slash `/`. - A default status code of `301` is applied if no [status code](#http-status-codes) is provided. -- The `_redirects` file has a file size limit of 64KB and a maximum of 1,000 rules per project. - Only the first 1,000 rules are processed. +- The `_redirects` file has a file size limit and a maximum number of rules per project, + configured at the instance level. Only the first matching rules within the configured maximum are processed. + The default file size limit is 64KB, and the default maximum number of rules is 1,000. - If your GitLab Pages site uses the default domain name (such as `namespace.gitlab.io/projectname`) you must prefix every rule with the project name: diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index d5a7058d3d2..96e51b061ee 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -55,7 +55,6 @@ threads. Some quick actions might not be available to all subscription tiers. | `/assign me` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Assign yourself. | | `/assign_reviewer @user1 @user2` or `/reviewer @user1 @user2` or `/request_review @user1 @user2` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Assign one or more users as reviewers. | | `/assign_reviewer me` or `/reviewer me` or `/request_review me` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Assign yourself as a reviewer. | -| `/attention @user1` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | [Request attention](merge_requests/index.md#request-attention-to-a-merge-request) to a merge request from a user. | | `/award :emoji:` | **{check-circle}** Yes | **{check-circle}** Yes | **{check-circle}** Yes | Toggle emoji award. | | `/child_epic <epic>` | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Add child epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7330) in GitLab 12.0). | | `/clear_health_status` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Clear [health status](issues/managing_issues.md#health-status) ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213814) in GitLab 14.7). | diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index d5ddc0468e1..1d448ca5c94 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -210,7 +210,7 @@ In the second workflow, the `release` job runs in multiple pipelines. To prevent ```yaml release_job: rules: - - if: $CI_COMMIT_TAG + - if: $CI_COMMIT_TAG when: never # Do not run this job in a tag pipeline - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch script: @@ -317,6 +317,25 @@ You can edit the release title, notes, associated milestones, and asset links. To change the release date use the [Releases API](../../../api/releases/index.md#update-a-release). +## Delete a release + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213862) in GitLab 15.2 + +When you delete a release, its assets are also deleted. However, the associated +Git tag is not deleted. + +Prerequisites: + +- You must have at least the Developer role. Read more about [Release permissions](#release-permissions). + +To delete a release in the UI: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Releases**. +1. In the top-right corner of the release you want to delete, select **Edit this release** (**{pencil}**). +1. On the **Edit Release** page, select **Delete**. +1. Select **Delete release**. + ## Associate milestones with a release > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29020) in GitLab 12.5. diff --git a/doc/user/project/repository/branches/default.md b/doc/user/project/repository/branches/default.md index e087ed6c439..747da817195 100644 --- a/doc/user/project/repository/branches/default.md +++ b/doc/user/project/repository/branches/default.md @@ -76,7 +76,7 @@ overrides it. ### Group-level custom initial branch name -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221014) in GitLab 13.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221014) in GitLab 13.6. Users with at least the Owner role of groups and subgroups can configure the default branch name for a group: diff --git a/doc/user/project/repository/forking_workflow.md b/doc/user/project/repository/forking_workflow.md index 0e6c98457c7..85bea80f777 100644 --- a/doc/user/project/repository/forking_workflow.md +++ b/doc/user/project/repository/forking_workflow.md @@ -68,4 +68,4 @@ changes are added to the repository and branch you're merging into. ## Removing a fork relationship -You can unlink your fork from its upstream project in the [advanced settings](../settings/index.md#removing-a-fork-relationship). +You can unlink your fork from its upstream project in the [advanced settings](../settings/index.md#remove-a-fork-relationship). diff --git a/doc/user/project/repository/img/repository_languages_v12_2.gif b/doc/user/project/repository/img/repository_languages_v12_2.gif Binary files differdeleted file mode 100644 index d0a0e57c919..00000000000 --- a/doc/user/project/repository/img/repository_languages_v12_2.gif +++ /dev/null diff --git a/doc/user/project/repository/img/repository_languages_v15_2.png b/doc/user/project/repository/img/repository_languages_v15_2.png Binary files differnew file mode 100644 index 00000000000..94cfa1cc161 --- /dev/null +++ b/doc/user/project/repository/img/repository_languages_v15_2.png diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index 02b5639cae8..a8937d4f705 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -116,7 +116,7 @@ You can download the source code that's stored in a repository. For the default branch of each repository, GitLab determines which programming languages are used. This information is displayed on the **Project information** page. -![Repository Languages bar](img/repository_languages_v12_2.gif) +![Repository Languages bar](img/repository_languages_v15_2.png) When new files are added, this information can take up to five minutes to update. @@ -232,7 +232,7 @@ When a repository path changes, GitLab handles the transition from the old location to the new one with a redirect. When you [rename a user](../../profile/index.md#change-your-username), -[change a group path](../../group/index.md#change-a-groups-path), or [rename a repository](../settings/index.md#renaming-a-repository): +[change a group path](../../group/index.md#change-a-groups-path), or [rename a repository](../settings/index.md#rename-a-repository): - URLs for the namespace and everything under it, like projects, are redirected to the new URLs. diff --git a/doc/user/project/repository/managing_large_repositories.md b/doc/user/project/repository/managing_large_repositories.md index 76f66f41297..93b94ac0641 100644 --- a/doc/user/project/repository/managing_large_repositories.md +++ b/doc/user/project/repository/managing_large_repositories.md @@ -16,7 +16,7 @@ On this page we detail several best practices to improve performance with these It's *strongly* recommended in any Git system that binary or blob files (for example, packages, audio, video, graphics, etc.) are stored as Large File Storage (LFS) objects. In such setup, the Objects are stored elsewhere, such as in Object Storage, and this can reduce the repository size significantly, thus improving performance. -Refer to the [Git LFS docs for more information](../../../topics/git/lfs/index.md). +Refer to the [Git LFS documentation for more information](../../../topics/git/lfs/index.md). ## Gitaly Pack Objects Cache @@ -34,7 +34,7 @@ In these types of setups it's recommended that the GitLab environment used match Gitaly Cluster can notably improve large repository performance as it holds multiple replicas of the repository across several nodes. As a result, Gitaly Cluster can load balance read requests against those repositories and is also fault tolerant. -It's recommended for large repositories, however, Gitaly Cluster is a large solution with additional complexity of setup and management. Refer to the [Gitaly Cluster docs for more information](../../../administration/gitaly/index.md), specifically the [Before deploying Gitaly Cluster](../../../administration/gitaly/index.md#before-deploying-gitaly-cluster) section. +It's recommended for large repositories, however, Gitaly Cluster is a large solution with additional complexity of setup, and management. Refer to the [Gitaly Cluster documentation for more information](../../../administration/gitaly/index.md), specifically the [Before deploying Gitaly Cluster](../../../administration/gitaly/index.md#before-deploying-gitaly-cluster) section. ## Keep GitLab up to date @@ -46,6 +46,6 @@ Large repositories tend to be monorepos. This in turn typically means that these CI/CD loads tend to be concurrent as pipelines are scheduled during set times. As a result, the Git requests against the repositories can spike notably during these times and lead to reduced performance for both CI and users alike. -When designing CI/CD pipelines, it's advisable to reduce their concurrency by staggering them to run at different times, for example, a set running at one time and then another set running several minutes later. +When designing CI/CD pipelines, it's advisable to reduce their concurrency by staggering them to run at different times, for example, a set running at one time, and another set running several minutes later. -There's several other actions that can be explored to improve CI/CD performance with large repositories. Refer to the [Runner docs for more information](../../../ci/large_repositories/index.md). +There's several other actions that can be explored to improve CI/CD performance with large repositories. Refer to the [Runner documentation for more information](../../../ci/large_repositories/index.md). diff --git a/doc/user/project/repository/mirror/index.md b/doc/user/project/repository/mirror/index.md index fe1c9653cfe..4537f8520cd 100644 --- a/doc/user/project/repository/mirror/index.md +++ b/doc/user/project/repository/mirror/index.md @@ -8,7 +8,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/repository_mirroring.htm # Repository mirroring **(FREE)** You can _mirror_ a repository to and from external sources. You can select which repository -serves as the source. Branches, tags, and commits can be mirrored. +serves as the source. Branches, tags, and commits are synced automatically. NOTE: SCP-style URLs are **not** supported. However, the work for implementing SCP-style URLs is tracked @@ -302,3 +302,12 @@ fail nor succeed. They also do not leave a clear log. To check for this problem: 1. After you run the command, the [background jobs page](../../../admin_area/index.md#background-jobs) should show new mirroring jobs being scheduled, especially when [triggered manually](#update-a-mirror). + +### Invalid URL + +If you receive this error while setting up mirroring over [SSH](#ssh-authentication), make sure the URL is in a valid format. + +Mirroring does not support the short version of SSH clone URLs (`git@gitlab.com:gitlab-org/gitlab.git`) +and requires the full version including the protocol (`ssh://git@gitlab.com/gitlab-org/gitlab.git`). + +Make sure that host and project path are separated using `/` instead of `:`. diff --git a/doc/user/project/repository/mirror/pull.md b/doc/user/project/repository/mirror/pull.md index 3599faf4de6..88104e34eb4 100644 --- a/doc/user/project/repository/mirror/pull.md +++ b/doc/user/project/repository/mirror/pull.md @@ -97,7 +97,7 @@ assigned when you set up pull mirroring. Pull mirroring uses polling to detect new branches and commits added upstream, often minutes afterwards. You can notify GitLab using an -[API call](../../../../api/projects.md#start-the-pull-mirroring-process-for-a-project), +[API call](../../../../api/projects.md#start-the-pull-mirroring-process-for-a-project), but the [minimum interval for pull mirroring limits](index.md#force-an-update) is still enforced. For more information, read diff --git a/doc/user/project/repository/reducing_the_repo_size_using_git.md b/doc/user/project/repository/reducing_the_repo_size_using_git.md index 83fafd409e8..b0ae1b7d1e0 100644 --- a/doc/user/project/repository/reducing_the_repo_size_using_git.md +++ b/doc/user/project/repository/reducing_the_repo_size_using_git.md @@ -1,8 +1,7 @@ --- -stage: Create +stage: Systems group: Gitaly info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Reduce repository size **(FREE)** @@ -20,7 +19,7 @@ over [`git filter-branch`](https://git-scm.com/docs/git-filter-branch) and WARNING: Rewriting repository history is a destructive operation. Make sure to back up your repository before -you begin. The best way back up a repository is to +you begin. The best way to back up a repository is to [export the project](../settings/import_export.md#export-a-project-and-its-data). ## Purge files from repository history @@ -36,6 +35,11 @@ other internal references (refs) that are automatically created by GitLab. These These refs are not automatically downloaded and hidden refs are not advertised, but we can remove these refs using a project export. +WARNING: +This process is not suitable for removing sensitive data like password or keys from your repository. +Information about commits, including file content, is cached in the database, and remain +visible even after they have been removed from the repository. + To purge files from a GitLab repository: 1. Install either [`git filter-repo`](https://github.com/newren/git-filter-repo/blob/main/INSTALL.md) or @@ -248,11 +252,6 @@ increased, your only option is to: 1. Prune all the unneeded stuff locally. 1. Create a new project on GitLab and start using that instead. -WARNING: -This process is not suitable for removing sensitive data like password or keys from your repository. -Information about commits, including file content, is cached in the database, and remain -visible even after they have been removed from the repository. - ## Troubleshooting ### Incorrect repository statistics shown in the GUI diff --git a/doc/user/project/repository/web_editor.md b/doc/user/project/repository/web_editor.md index 370a349b982..4ca341f0535 100644 --- a/doc/user/project/repository/web_editor.md +++ b/doc/user/project/repository/web_editor.md @@ -137,7 +137,7 @@ The **Create merge request** button doesn't display if: - Your project has an active fork relationship. To make this button appear, one possible workaround is to -[remove your project's fork relationship](../settings/index.md#removing-a-fork-relationship). +[remove your project's fork relationship](../settings/index.md#remove-a-fork-relationship). After removal, the fork relationship cannot be restored. This project can no longer be able to receive or send merge requests to the source project, or other forks. diff --git a/doc/user/project/settings/img/cve_id_request_toggle.png b/doc/user/project/settings/img/cve_id_request_toggle.png Binary files differdeleted file mode 100644 index 53ec804922c..00000000000 --- a/doc/user/project/settings/img/cve_id_request_toggle.png +++ /dev/null diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 46a6c1a049e..7d1bfcaab59 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -278,24 +278,34 @@ When you disable a feature, the following additional features are also disabled: - Metrics dashboard access requires reading project environments and deployments. Users with access to the metrics dashboard can also access environments and deployments. -## Disabling the CVE ID request button **(FREE SAAS)** +## Disable CVE identifier request in issues **(FREE SAAS)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41203) in GitLab 13.4, only for public projects on GitLab.com. -In applicable environments, a [**Create CVE ID Request** button](../../application_security/cve_id_request.md) -is present in the issue sidebar. The button may be disabled on a per-project basis by toggling the -setting **Enable CVE ID requests in the issue sidebar**. +In some environments, users can submit a [CVE identifier request](../../application_security/cve_id_request.md) in an issue. -![CVE ID Request toggle](img/cve_id_request_toggle.png) +To disable the CVE identifier request option in issues in your project: -## Disabling email notifications +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Visibility, project features, permissions** section. +1. Under **Issues**, turn off the **CVE ID requests in the issue sidebar** toggle. +1. Select **Save changes**. + +## Disable project email notifications -Project owners can disable all [email notifications](../../profile/notifications.md) -related to the project by selecting the **Disable email notifications** checkbox. +Prerequisites: + +- You must be an Owner of the project to disable email notifications related to the project. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Visibility, project features, permissions** section. +1. Clear the **Disable email notifications** checkbox. ## Configure merge request settings for a project -Set up your project's merge request settings: +Configure your project's merge request settings: - Set up the [merge request method](../merge_requests/methods/index.md) (merge commit, fast-forward merge). - Add merge request [description templates](../description_templates.md#description-templates). @@ -317,91 +327,74 @@ Enable [Service Desk](../service_desk.md) for your project to offer customer sup Learn how to [export a project](import_export.md#import-a-project-and-its-data) in GitLab. -## Advanced settings +## Advanced project settings -Here you can run housekeeping, archive, rename, transfer, -[remove a fork relationship](#removing-a-fork-relationship), or delete a project. +Use the advanced settings to archive, rename, transfer, +remove a fork relationship, or delete a project. -## Archiving a project +### Archive a project -Archiving a project makes it read-only for all users and indicates that it's -no longer actively maintained. Projects that have been archived can also be -unarchived. Only project owners and administrators have the -[permissions](../../permissions.md#project-members-permissions) to archive a project. - -When a project is archived, the repository, packages, issues, merge requests, and all -other features are read-only. Archived projects are also hidden -in project listings. +When you archive a project, the repository, packages, issues, merge requests, and all +other features are read-only. Archived projects are also hidden from project listings. To archive a project: -1. Navigate to your project's **Settings > General**. -1. Under **Advanced**, select **Expand**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Advanced**. 1. In the **Archive project** section, select **Archive project**. -1. Confirm the action when asked to. - -## Unarchiving a project +1. To confirm, select **OK**. -Unarchiving a project removes the read-only restriction on a project, and makes it -available in project listings. Only project owners and administrators have the -[permissions](../../permissions.md#project-members-permissions) to unarchive a project. +### Unarchive a project -To find an archived project: +When you unarchive a project, you remove the read-only restriction and make it +available in project lists. -1. Sign in to GitLab as the project owner or a user with administrator access. -1. If you: - - Have the project's URL, open the project's page in your browser. - - Don't have the project's URL: - 1. On the top bar, select **Menu > Project**. - 1. Select **Explore projects**. - 1. In the **Sort projects** dropdown box, select **Show archived projects**. - 1. In the **Filter by name** field, provide the project's name. - 1. Select the link to the project to open its **Details** page. +Prerequisites: -Next, to unarchive the project: +- To unarchive a project, you must be an administrator or a project Owner. -1. Navigate to your project's **Settings > General**. +1. Find the archived project. + 1. On the top bar, select **Menu > Project**. + 1. Select **Explore projects**. + 1. In the **Sort projects** dropdown list, select **Show archived projects**. + 1. In the **Filter by name** field, enter the project name. + 1. Select the project link. +1. On the left sidebar, select **Settings > General**. 1. Under **Advanced**, select **Expand**. 1. In the **Unarchive project** section, select **Unarchive project**. -1. Confirm the action when asked to. +1. To confirm, select **OK**. -## Renaming a repository +### Rename a repository -NOTE: -Only project maintainers and administrators have the [permissions](../../permissions.md#project-members-permissions) to rename a -repository. Not to be confused with a project's name where it can also be -changed from the [general project settings](#edit-project-name-and-description). - -A project's repository name defines its URL (the one you use to access the -project via a browser) and its place on the file disk where GitLab is installed. +A project's repository name defines its URL and its place on the file disk +where GitLab is installed. -To rename a repository: +Prerequisites: -1. Navigate to your project's **Settings > General**. -1. Under **Advanced**, select **Expand**. -1. Under **Change path**, update the repository's path. -1. Select **Change path**. +You must be a project maintainer or administrator to rename a repository. -Remember that this can have unintended side effects since everyone with the -old URL can't push or pull. Read more about what happens with the +NOTE: +When you change the repository path, users may experience issues if they push to, or pull from, the old URL. For more information, see [redirects when renaming repositories](../repository/index.md#what-happens-when-a-repository-path-changes). -## Transferring an existing project into another namespace +To rename a repository: -NOTE: -Only project owners and administrators have the [permissions](../../permissions.md#project-members-permissions) -to transfer a project. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Advanced** section. +1. In the **Change path** text box, edit the path. +1. Select **Change path**. + +## Transfer a project to another namespace -You can transfer an existing project to another [group](../../group/index.md), -or you can transfer a [personal project](../working_with_projects.md#view-personal-projects) to a group. +When you transfer a project to another namespace, you move the project to a different group. Prerequisites: -- A group for your project. You can [view your existing groups](../../group/index.md#view-groups) - to find a suitable group. If you don't have a group, [create one](../../group/index.md#create-a-group). -- You must have at least the Maintainer role in that group. -- You must be the Owner of that project. -- The group to which the project is being transferred to must allow creation of new projects. +- You must have at least the Maintainer role for the [group](../../group/index.md#create-a-group) to which you are transferring. +- You must be the Owner of the project you transfer. +- The group must allow creation of new projects. - The project must not contain any [container images](../../packages/container_registry/index.md#limitations). - If you transfer a project to a different root namespace, the project must not contain any @@ -416,19 +409,18 @@ To transfer a project: 1. Select **Transfer project**. 1. Enter the project's name and select **Confirm**. -You are redirected to the project's new URL. Read what happens with the -[redirects from the old URL to the new one](../repository/index.md#what-happens-when-a-repository-path-changes). +You are redirected to the project's new page and GitLab applies a redirect. For more information about repository redirects, see [What happens when a repository path changes](../repository/index.md#what-happens-when-a-repository-path-changes). NOTE: -GitLab administrators can use the [administration interface](../../admin_area/index.md#administering-projects) -to move any project to any namespace if needed. +If you are an administrator, you can also use the [administration interface](../../admin_area/index.md#administering-projects) +to move any project to any namespace. -## Transferring a GitLab.com project to a different subscription tier +### Transferring a GitLab SaaS project to a different subscription tier -When you transfer a project from a namespace that's licensed for GitLab SaaS Premium or Ultimate to Free, some data related to the paid features is deleted. +When you transfer a project from a namespace licensed for GitLab SaaS Premium or Ultimate to GitLab Free, the following paid feature data is deleted: -For example, [project access tokens](../../../user/project/settings/project_access_tokens.md) are revoked, and -[pipeline subscriptions](../../../ci/pipelines/multi_project_pipelines.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt) +- [Project access tokens](../../../user/project/settings/project_access_tokens.md) are revoked +- [Pipeline subscriptions](../../../ci/pipelines/multi_project_pipelines.md#trigger-a-pipeline-when-an-upstream-project-is-rebuilt) and [test cases](../../../ci/test_cases/index.md) are deleted. ## Delete a project @@ -460,7 +452,7 @@ in GitLab 12.6, and then to [immediate deletion](https://gitlab.com/gitlab-org/g Projects can be deleted after a delay period. Multiple settings can affect whether delayed project deletion is enabled for a particular project: -- Self-managed instance [settings](../../admin_area/settings/visibility_and_access_controls.md#deletion-protection). +- Self-managed instance [settings](../../admin_area/settings/visibility_and_access_controls.md#delayed-project-deletion). You can enable delayed project deletion as the default setting for new groups, and configure the number of days for the delay. For GitLab.com, see the [GitLab.com settings](../../gitlab_com/index.md#delayed-project-deletion). - Group [settings](../../group/index.md#enable-delayed-project-deletion) to enabled delayed project deletion for all @@ -499,27 +491,23 @@ To restore a project marked for deletion: 1. Navigate to your project, and select **Settings > General > Advanced**. 1. In the Restore project section, select **Restore project**. -## Removing a fork relationship +## Remove a fork relationship + +Prerequisites: -Forking is a great way to [contribute to a project](../repository/forking_workflow.md) -of which you're not a member. -If you want to use the fork for yourself and don't need to send -[merge requests](../merge_requests/index.md) to the upstream project, -you can safely remove the fork relationship. +- You must be a project owner to remove a fork relationship. WARNING: -Once removed, you can't send merge requests to the source, and if anyone has forked your project, their fork also loses the relationship. +If you remove a fork relationship, you can't send merge requests to the source. If anyone has forked your project, their fork also loses the relationship. To restore the fork relationship, [use the API](../../../api/projects.md#create-a-forked-fromto-relation-between-existing-projects). -To do so: +To remove a fork relationship: -1. Navigate to your project's **Settings > General > Advanced**. -1. Under **Remove fork relationship**, select the likewise-labeled button. -1. Confirm the action by typing the project's path as instructed. - -NOTE: -Only project owners have the [permissions](../../permissions.md#project-members-permissions) -to remove a fork relationship. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General**. +1. Expand **Advanced**. +1. In the **Remove fork relationship** section, select **Remove fork relationship**. +1. To confirm, enter the project path and select **Confirm**. ## Monitor settings diff --git a/doc/user/project/wiki/img/content_editor_v14.6.png b/doc/user/project/wiki/img/content_editor_v14.6.png Binary files differdeleted file mode 100644 index 55fca0ace1e..00000000000 --- a/doc/user/project/wiki/img/content_editor_v14.6.png +++ /dev/null diff --git a/doc/user/project/wiki/img/use_new_editor_button_v14.6.png b/doc/user/project/wiki/img/use_new_editor_button_v14.6.png Binary files differdeleted file mode 100644 index 078fed8a1e9..00000000000 --- a/doc/user/project/wiki/img/use_new_editor_button_v14.6.png +++ /dev/null diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index 5ae0cf46d9b..6e320923496 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -329,16 +329,15 @@ to disable the wiki but toggle it on (in blue). > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345398) switching between editing experiences in GitLab 14.7 [with a flag](../../../administration/feature_flags.md) named `wiki_switch_between_content_editor_raw_markdown`. Enabled by default. > - Switching between editing experiences generally available in GitLab 14.10. [Feature flag `wiki_switch_between_content_editor_raw_markdown`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/83760) removed. -GitLab version 14.0 introduces a WYSIWYG editing experience for GitLab Flavored Markdown -in Wikis through the [Content Editor](../../../development/fe_guide/content_editor.md). -The Content Editor is under active development, and is not yet the default editing -experience in the Wiki. To opt in for the new editor: +GitLab provides a WYSIWYG editing experience for GitLab Flavored Markdown in wikis. -1. Create a new wiki page, or edit an existing one. -1. Ensure the wiki page uses the Markdown format. Other formats are not yet supported. -1. Above the content field, select **Edit rich text**: +Support includes: - ![Use new editor button image](img/use_new_editor_button_v14.6.png) +- Text formatting options, including bold, italics, block quotes, headings, and inline code. +- List formatting for unordered, numbered, and checklists. +- Creating and editing the structure of tables. +- Inserting and formatting code blocks with syntax highlighting. +- Live preview of Mermaid, PlantUML, and Kroki diagrams ([Introduced]<https://gitlab.com/gitlab-org/gitlab/-/merge_requests/86701> in GitLab 15.2). ### Use the Content Editor @@ -346,9 +345,10 @@ experience in the Wiki. To opt in for the new editor: 1. Select **Markdown** as your format. 1. Above **Content**, select **Edit rich text**. 1. Customize your page's content using the various formatting options available in the content editor. -1. Select **Create page** for a new page, or **Save changes** for an existing page: +1. Select **Create page** for a new page, or **Save changes** for an existing page. - ![Content Editor in Wikis image](img/content_editor_v14.6.png) +The rich text editing mode remains the default until you switch back to +[edit the raw source](#switch-back-to-the-old-editor). ### Switch back to the old editor diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index 83cab819f54..9572bc241fc 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -198,7 +198,7 @@ GitLab creates your project in your chosen namespace. You cannot use `git push` to create projects with project paths that: - Have previously been used. -- Have been [renamed](settings/index.md#renaming-a-repository). +- Have been [renamed](settings/index.md#rename-a-repository). Previously used project paths have a redirect. The redirect causes push attempts to redirect requests to the renamed project location, instead of creating a new project. To create a new project for a previously @@ -391,7 +391,7 @@ To use a project as a Go package, use the `go get` and `godoc.org` discovery req Prerequisites: - Your GitLab instance must be accessible with HTTPS. -- You must have a [personal access token](../profile/personal_access_tokens.md). +- You must have a [personal access token](../profile/personal_access_tokens.md) with `read_api` scope. To authenticate Go requests, create a [`.netrc`](https://everything.curl.dev/usingcurl/netrc) file with the following information: @@ -423,7 +423,7 @@ Configure Git to either: - Use SSH instead of HTTPS: ```shell - git config --global url."git@gitlab.example.com".insteadOf "https://gitlab.example.com" + git config --global url."git@gitlab.example.com:".insteadOf "https://gitlab.example.com/" ``` ### Disable Go module fetching for private projects diff --git a/doc/user/public_access.md b/doc/user/public_access.md index cca753a2830..d821c1abe47 100644 --- a/doc/user/public_access.md +++ b/doc/user/public_access.md @@ -70,6 +70,8 @@ Prerequisite: Prerequisite: - You must have the Owner role for a group. +- Subgroups and projects must already have visibility settings that are at least as + restrictive as the new setting for the group. 1. On the top bar, select **Menu > Groups** and find your project. 1. On the left sidebar, select **Settings > General**. diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index 075c9b6154b..90d6a15901a 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -84,15 +84,8 @@ its performance: ## Global Search validation -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/346263) in GitLab 14.6 [with a flag](../../administration/feature_flags.md) named `prevent_abusive_searches`. Disabled by default. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `prevent_abusive_searches`. -The feature is not ready for production use. - To prevent abusive searches, such as searches that may result in a Distributed Denial of Service (DDoS), Global Search ignores, logs, and -doesn't return any results for searches considered abusive according to the following criteria, if `prevent_abusive_searches` feature flag is enabled: +doesn't return any results for searches considered abusive according to the following criteria: - Searches with less than 2 characters. - Searches with any term greater than 100 characters. URL search terms have a maximum of 200 characters. @@ -101,8 +94,7 @@ doesn't return any results for searches considered abusive according to the foll - Searches with a `repository_ref` or `project_ref` parameter that has special characters not allowed by [Git refname](https://git-scm.com/docs/git-check-ref-format). - Searches with a `scope` that is unknown. -Regardless of the status of the `prevent_abusive_searches` feature flag, searches that don't -comply with the criteria described below aren't logged as abusive but are flagged with an error: +Searches that don't comply with the criteria described below aren't logged as abusive but are flagged with an error: - Searches with more than 4096 characters. - Searches with more than 64 terms. diff --git a/doc/user/search/img/basic_search_results_v15_1.png b/doc/user/search/img/basic_search_results_v15_1.png Binary files differindex b85627c9b95..0de0b976d7d 100644 --- a/doc/user/search/img/basic_search_results_v15_1.png +++ b/doc/user/search/img/basic_search_results_v15_1.png diff --git a/doc/user/search/img/code_search_git_blame_v15_1.png b/doc/user/search/img/code_search_git_blame_v15_1.png Binary files differindex e61ee5993c2..426f829b186 100644 --- a/doc/user/search/img/code_search_git_blame_v15_1.png +++ b/doc/user/search/img/code_search_git_blame_v15_1.png diff --git a/doc/user/ssh.md b/doc/user/ssh.md index 27bb7124afe..e884d762379 100644 --- a/doc/user/ssh.md +++ b/doc/user/ssh.md @@ -354,7 +354,7 @@ can do this by using the command in the [previous topic](#use-different-keys-for However, even if you set `IdentitiesOnly` to `yes`, you cannot sign in if an `IdentityFile` exists outside of a `Host` block. -Instead, you can assign aliases to hosts in the `~.ssh/config` file. +Instead, you can assign aliases to hosts in the `~/.ssh/config` file. - For the `Host`, use an alias like `user_1.gitlab.com` and `user_2.gitlab.com`. Advanced configurations diff --git a/doc/user/tasks.md b/doc/user/tasks.md index fc49661c61c..36236f2969e 100644 --- a/doc/user/tasks.md +++ b/doc/user/tasks.md @@ -6,7 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Tasks **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334812) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `work_items`. Disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334812) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `work_items`. Disabled by default. +> - [Creating, editing, and deleting tasks](https://gitlab.com/groups/gitlab-org/-/epics/7169) introduced in GitLab 15.0. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, @@ -32,8 +33,7 @@ to work items and adding custom work item types, visit To create a task: 1. In an issue description, create a [task list](markdown.md#task-lists). -1. Hover over a task item and select **Convert to work item** (**{doc-new}**). -1. Confirm or edit the title, and select **Create work item**. +1. Hover over a task item and select **Create task** (**{doc-new}**). ## Edit a task @@ -42,11 +42,11 @@ To edit a task: 1. In the issue description, view the task links. 1. Select a link. The task is displayed. - To edit the description, select **Edit**, then select **Save**. - - To edit the title or state, make your changes, then click outside the field. The changes are saved automatically. + - To edit the title or state, make your changes, then select any area outside the field. The changes are saved automatically. ## Delete a task To delete a task: 1. In the issue description, select the task. -1. From the options menu (**{ellipsis_v}**), select **Delete work item**. +1. From the options menu (**{ellipsis_v}**), select **Delete task**. diff --git a/doc/user/usage_quotas.md b/doc/user/usage_quotas.md index 84c98a60917..c863a9d8270 100644 --- a/doc/user/usage_quotas.md +++ b/doc/user/usage_quotas.md @@ -10,11 +10,57 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/13294) in GitLab 12.0. > - Moved to GitLab Free. -NOTE: -Free tier namespaces on GitLab SaaS have a 5GB storage limit. This limit is not visible on the storage quota page nor currently enforced for users who exceed the limit. To learn more, visit our [pricing page](https://about.gitlab.com/pricing/). +## Namespace storage limit -A project's repository has a free storage quota of 10 GB. When a project's repository reaches -the quota it is locked. You cannot push changes to a locked project. To monitor the size of each +Namespaces on a GitLab SaaS Free tier have a 5 GB storage limit. For more information, see our [pricing page](https://about.gitlab.com/pricing/). +This limit is not visible on the storage quota page, but we plan to make it visible and enforced starting October 19, 2022. + +Storage types that add to the total namespace storage are: + +- Git repository +- Git LFS +- Artifacts +- Container registry +- Package registry +- Dependecy proxy +- Wiki +- Snippets + +If your total namespace storage exceeds the available namespace storage quota, all projects under the namespace are locked. A locked project will not be able to push to the repository, run pipelines and jobs, or build and push packages. + +To prevent exceeding the namespace storage quota, you can: + +1. [Purchase more storage](../subscriptions/gitlab_com/index.md#purchase-more-storage-and-transfer). +1. [Upgrade to a paid tier](../subscriptions/gitlab_com/#upgrade-your-gitlab-saas-subscription-tier). +1. [Reduce storage usage](#manage-your-storage-usage). + +### Namespace storage limit enforcement schedule + +Starting October 19, 2022, a storage limit will be enforced on all GitLab Free namespaces. +We will start with a large limit enforcement and eventually reduce it to 5 GB. + +The following table describes the enforcement schedule, which is subject to change. + +| Target enforcement date | Limit | Expected Impact | Status | +| ------ | ------ | ------ | ------ | +| October 19, 2022 | 45,000 GB | LOW | Pending (**{hourglass}**)| +| October 20, 2022 | 7,500 GB | LOW | Pending (**{hourglass}**)| +| October 24, 2022 | 500 GB | MEDIUM | Pending (**{hourglass}**)| +| October 27, 2022 | 75 GB | MEDIUM HIGH | Pending (**{hourglass}**)| +| November 2, 2022 | 10 GB | HIGH | Pending (**{hourglass}**)| +| November 9, 2022 | 5 GB | VERY HIGH | Pending (**{hourglass}**)| + +Namespaces that reach the enforced limit will have their projects locked. To unlock your project, you will have to [manage its storage](#manage-your-storage-usage). + +### Project storage limit + +Namespaces on a GitLab SaaS **paid** tier (Premium and Ultimate) have a storage limit on their project repositories. +A project's repository has a storage quota of 10 GB. A namespace has either a namespace-level storage limit or a project-level storage limit, but not both. + +- Paid tier namespaces have project-level storage limits enforced. +- Free tier namespaces have namespace-level storage limits. + +When a project's repository reaches the quota, the project is locked. You cannot push changes to a locked project. To monitor the size of each repository in a namespace, including a breakdown for each project, you can [view storage usage](#view-storage-usage). To allow a project's repository to exceed the free quota you must purchase additional storage. For more details, see [Excess storage usage](#excess-storage-usage). |