diff options
| author | GitLab Bot <gitlab-bot@gitlab.com> | 2021-11-18 16:16:36 +0300 |
|---|---|---|
| committer | GitLab Bot <gitlab-bot@gitlab.com> | 2021-11-18 16:16:36 +0300 |
| commit | 311b0269b4eb9839fa63f80c8d7a58f32b8138a0 (patch) | |
| tree | 07e7870bca8aed6d61fdcc810731c50d2c40af47 /doc | |
| parent | 27909cef6c4170ed9205afa7426b8d3de47cbb0c (diff) | |
Add latest changes from gitlab-org/gitlab@14-5-stable-eev14.5.0-rc42
Diffstat (limited to 'doc')
589 files changed, 15308 insertions, 8578 deletions
diff --git a/doc/.vale/gitlab/Acronyms.yml b/doc/.vale/gitlab/Acronyms.yml index b75f81dd075..23285fd0038 100644 --- a/doc/.vale/gitlab/Acronyms.yml +++ b/doc/.vale/gitlab/Acronyms.yml @@ -55,6 +55,7 @@ exceptions: - FAQ - FIFO - FIPS + - FLAG - FOSS - FQDN - FREE @@ -140,6 +141,7 @@ exceptions: - RPM - RPS - RSA + - RDS - RSS - RVM - SAAS diff --git a/doc/.vale/gitlab/British.yml b/doc/.vale/gitlab/British.yml index 152723ead26..f724eb19fa9 100644 --- a/doc/.vale/gitlab/British.yml +++ b/doc/.vale/gitlab/British.yml @@ -19,6 +19,10 @@ swap: analyse: analyze annexe: annex apologise: apologize + authorise: authorize + authorised: authorized + authorisation: authorization + authorising: authorizing behaviour: behavior busses: buses calibre: caliber diff --git a/doc/.vale/gitlab/Substitutions.yml b/doc/.vale/gitlab/Substitutions.yml index e6c150fb8be..dde05b993ec 100644 --- a/doc/.vale/gitlab/Substitutions.yml +++ b/doc/.vale/gitlab/Substitutions.yml @@ -41,9 +41,15 @@ swap: developer access: the Developer role developer permission: the Developer role developer permissions: the Developer role + guest access: the Guest role + guest permission: the Guest role + guest permissions: the Guest role maintainer access: the Maintainer role maintainer permission: the Maintainer role maintainer permissions: the Maintainer role owner access: the Owner role owner permission: the Owner role owner permissions: the Owner role + reporter access: the Reporter role + reporter permission: the Reporter role + reporter permissions: the Reporter role diff --git a/doc/.vale/gitlab/spelling-exceptions.txt b/doc/.vale/gitlab/spelling-exceptions.txt index d397a436ff9..df228e61278 100644 --- a/doc/.vale/gitlab/spelling-exceptions.txt +++ b/doc/.vale/gitlab/spelling-exceptions.txt @@ -45,8 +45,8 @@ autoscales autoscaling awardable awardables -Ayoa Axios +Ayoa Azure B-tree backfilling @@ -93,9 +93,9 @@ canonicalized captcha CentOS Certbot +chai changeset changesets -chai ChaosKube chatbot chatbots @@ -150,6 +150,7 @@ denormalized denormalizes denormalizing denylist +denylisted denylisting denylists deployer @@ -186,6 +187,7 @@ downvotes Dpl Dreamweaver Ecto +ElastiCache Elasticsearch enablement enqueued @@ -683,8 +685,8 @@ triaged triages triaging Trivy -truthy Truststore +truthy Twilio Twitter TypeScript diff --git a/doc/administration/audit_event_streaming.md b/doc/administration/audit_event_streaming.md new file mode 100644 index 00000000000..cee6cddbbf0 --- /dev/null +++ b/doc/administration/audit_event_streaming.md @@ -0,0 +1,70 @@ +--- +stage: Manage +group: Compliance +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Audit event streaming **(ULTIMATE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332747) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `ff_external_audit_events_namespace`. On GitLab.com, this feature is not available. +You should not use this feature for production environments. + +Event streaming allows owners of top-level groups to set an HTTP endpoint to receive **all** audit events about the group, and its +subgroups and projects. + +Top-level group owners can manage their audit logs in third-party systems such as Splunk, using the Splunk +[HTTP Event Collector](https://docs.splunk.com/Documentation/Splunk/8.2.2/Data/UsetheHTTPEventCollector). Any service that can receive +structured JSON data can be used as the endpoint. + +NOTE: +GitLab can stream a single event more than once to the same destination. Use the `id` key in the payload to deduplicate incoming data. + +## Add a new event streaming destination + +WARNING: +Event streaming destinations will receive **all** audit event data, which could include sensitive information. Make sure you trust the destination endpoint. + +To enable event streaming, a group owner must add a new event streaming destination using the `externalAuditEventDestinationCreate` mutation +in the GraphQL API. + +```graphql +mutation { + externalAuditEventDestinationCreate(input: { destinationUrl: "https://mydomain.io/endpoint/ingest", groupPath: "my-group" } ) { + errors + externalAuditEventDestination { + destinationUrl + group { + name + } + } + } +} +``` + +Event streaming is enabled if: + +- The returned `errors` object is empty. +- The API responds with `200 OK`. + +## List currently enabled streaming destinations + +Group owners can view a list of event streaming destinations at any time using the `externalAuditEventDesinations` query type. + +```graphql +query { + group(fullPath: "my-group") { + id + externalAuditEventDestinations { + nodes { + destinationUrl + id + } + } + } +} +``` + +If the resulting list is empty, then audit event streaming is not enabled for that group. diff --git a/doc/administration/audit_events.md b/doc/administration/audit_events.md index 572c341f2b2..2062016ef03 100644 --- a/doc/administration/audit_events.md +++ b/doc/administration/audit_events.md @@ -12,7 +12,10 @@ on a [paid plan](https://about.gitlab.com/pricing/). GitLab system administrators can also take advantage of the logs located on the file system. See [the logs system documentation](logs.md#audit_jsonlog) for more details. -You can generate an [Audit report](audit_reports.md) of audit events. +You can: + +- Generate an [audit report](audit_reports.md) of audit events. +- [Stream audit events](audit_event_streaming.md) to an external endpoint. ## Overview @@ -70,6 +73,17 @@ From there, you can see the following actions: - Group changed visibility. - User was added to group and with which [permissions](../user/permissions.md). - User sign-in via [Group SAML](../user/group/saml_sso/index.md). +- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8071) in GitLab 14.5, changes to the following + [group SAML](../user/group/saml_sso/index.md) configuration: + - Enabled status. + - Enforcing SSO-only authentication for web activity. + - Enforcing SSO-only authentication for Git and Dependency Proxy activity. + - Enforcing users to have dedicated group-managed accounts. + - Prohibiting outer forks. + - Identity provider SSO URL. + - Certificate fingerprint. + - Default membership role. + - SSO-SAML group sync configuration. - Permissions changes of a user assigned to a group. - Removed user from group. - Project repository imported into group. @@ -83,6 +97,7 @@ From there, you can see the following actions: - 2FA enforcement or grace period changed. - Roles allowed to create project changed. - Group CI/CD variable added, removed, or protected status changed. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30857) in GitLab 13.3. +- Compliance framework created, updated, or deleted. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340649) in GitLab 14.6. Group events can also be accessed via the [Group Audit Events API](../api/audit_events.md#group-audit-events) @@ -186,9 +201,16 @@ successful sign-in events: After upgrading from GitLab Free to a paid tier, successful sign-in events are the only Audit Events visible in Audit Events views until more events are logged. +### "Deleted User" events + +Audit events can be created for a user after the user is deleted. The user name associated with the event is set to +"Deleted User" because the actual user name is unknowable. For example, if a deleted user's access to a project is +removed automatically due to expiration, the audit event is created for "Deleted User". We are [investigating](https://gitlab.com/gitlab-org/gitlab/-/issues/343933) +whether this is avoidable. + ### Missing events -Some events are not tracked in Audit Events. See the following +Some events are not tracked in audit events. See the following epics for more detail on which events are not being tracked, and our progress on adding these events into GitLab: @@ -196,10 +218,12 @@ on adding these events into GitLab: - [Group settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/475) - [Instance-level settings and activity](https://gitlab.com/groups/gitlab-org/-/epics/476) -Don't see the event you want in any of the epics linked above? You can use the **Audit Event -Proposal** issue template to -[create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Audit%20Event%20Proposal) -to request it, or you can [add it yourself](../development/audit_event_guide/). +Don't see the event you want in any of the epics linked above? You can either: + +- Use the **Audit Event Proposal** issue template to + [create an issue](https://gitlab.com/gitlab-org/gitlab/-/issues/new?issuable_template=Audit%20Event%20Proposal) to + request it. +- [Add it yourself](../development/audit_event_guide/). ### Disabled events @@ -240,7 +264,7 @@ The search filters you can see depends on which audit level you are at. | Scope (Instance level) | A specific group, project, or user that the action was scoped to. | | Date range | Either via the date range buttons or pickers (maximum range of 31 days). Default is from the first day of the month to today's date. | - + ## Export to CSV **(PREMIUM SELF)** @@ -251,7 +275,7 @@ Export to CSV allows customers to export the current filter view of your audit e CSV file, which stores tabular data in plain text. The data provides a comprehensive view with respect to audit events. -To export the Audit Events to CSV: +To export the audit events to CSV: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Audit Events**. @@ -284,5 +308,5 @@ The first row contains the headers, which are listed in the following table alon ### Limitation -The Audit Events CSV file is limited to a maximum of `100,000` events. +The audit events CSV file is limited to a maximum of `100,000` events. The remaining records are truncated when this limit is reached. diff --git a/doc/administration/auth/atlassian.md b/doc/administration/auth/atlassian.md index b58bbfa8eac..14c48231a3d 100644 --- a/doc/administration/auth/atlassian.md +++ b/doc/administration/auth/atlassian.md @@ -41,7 +41,7 @@ To enable the Atlassian OmniAuth provider for passwordless authentication you mu sudo -u git -H editor /home/git/gitlab/config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings to enable single sign-on and add `atlassian_oauth2` as an OAuth provider. +1. See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings) for initial settings to enable single sign-on and add `atlassian_oauth2` as an OAuth provider. 1. Add the provider configuration for Atlassian: For Omnibus GitLab installations: diff --git a/doc/administration/auth/authentiq.md b/doc/administration/auth/authentiq.md index 835293ff500..19ee143a72a 100644 --- a/doc/administration/auth/authentiq.md +++ b/doc/administration/auth/authentiq.md @@ -27,7 +27,7 @@ Authentiq generates a Client ID and the accompanying Client Secret for you to us sudo -u git -H editor /home/git/gitlab/config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings to enable single sign-on and add Authentiq as an OAuth provider. +1. See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings) for initial settings to enable single sign-on and add Authentiq as an OAuth provider. 1. Add the provider configuration for Authentiq: diff --git a/doc/administration/auth/cognito.md b/doc/administration/auth/cognito.md index 41e77c10e27..d137489a838 100644 --- a/doc/administration/auth/cognito.md +++ b/doc/administration/auth/cognito.md @@ -40,7 +40,7 @@ The following steps enable AWS Cognito as an authentication provider: ## Configure GitLab -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings) for initial settings. 1. On your GitLab server, open the configuration file. **For Omnibus installations** @@ -88,4 +88,4 @@ Your sign-in page should now display a Cognito button below the regular sign-in To begin the authentication process, click the icon, and AWS Cognito asks the user to sign in and authorize the GitLab application. If successful, the user is redirected and signed in to your GitLab instance. -For more information, see the [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration). +For more information, see [Configure initial settings](../../integration/omniauth.md#configure-initial-settings). diff --git a/doc/administration/auth/crowd.md b/doc/administration/auth/crowd.md index f83710ef4c7..466e208a52e 100644 --- a/doc/administration/auth/crowd.md +++ b/doc/administration/auth/crowd.md @@ -36,7 +36,7 @@ this provider also allows Crowd authentication for Git-over-https requests. sudo -u git -H editor config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) +1. See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration: @@ -84,7 +84,7 @@ could not authorize you from Crowd because invalid credentials ``` Ensure the Crowd users who need to sign in to GitLab are authorized to the -[application](#configure-a-new-crowd-application) in the **Authorisation** step. +[application](#configure-a-new-crowd-application) in the **Authorization** step. This could be verified by trying "Authentication test" for Crowd (as of 2.11). - + diff --git a/doc/administration/auth/jwt.md b/doc/administration/auth/jwt.md index b74b70ee8c0..26e523cb802 100644 --- a/doc/administration/auth/jwt.md +++ b/doc/administration/auth/jwt.md @@ -25,7 +25,7 @@ JWT will provide you with a secret key for you to use. sudo -u git -H editor config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration. For Omnibus GitLab: diff --git a/doc/administration/auth/ldap/google_secure_ldap.md b/doc/administration/auth/ldap/google_secure_ldap.md index e5af8e8256a..69f0bfdce4c 100644 --- a/doc/administration/auth/ldap/google_secure_ldap.md +++ b/doc/administration/auth/ldap/google_secure_ldap.md @@ -35,7 +35,7 @@ The steps below cover: credentials' and 'Read user information'. Select 'Add LDAP Client' NOTE: - If you plan to use GitLab [LDAP Group Sync](index.md#group-sync) + If you plan to use GitLab [LDAP Group Sync](ldap_synchronization.md#group-sync) , turn on 'Read group information'.  diff --git a/doc/administration/auth/ldap/index.md b/doc/administration/auth/ldap/index.md index 92815f10b92..9047cfae1e9 100644 --- a/doc/administration/auth/ldap/index.md +++ b/doc/administration/auth/ldap/index.md @@ -19,58 +19,52 @@ This integration works with most LDAP-compliant directory servers, including: - Open LDAP. - 389 Server. -Users added through LDAP take a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users). +Users added through LDAP: -## Security +- Take a [licensed seat](../../../subscriptions/self_managed/index.md#billable-users). +- Can authenticate with Git using either their GitLab username or their email and LDAP password, + even if password authentication for Git + [is disabled](../../../user/admin_area/settings/sign_in_restrictions.md#password-authentication-enabled). -GitLab assumes that LDAP users: +The LDAP DN is associated with existing GitLab users when: -- Are not able to change their LDAP `mail`, `email`, or `userPrincipalName` attributes. - An LDAP user allowed to change their email on the LDAP server can potentially - [take over any account](#enable-ldap-sign-in-for-existing-gitlab-users) - on your GitLab server. -- Have unique email addresses. If not, it's possible for LDAP users with the same - email address to share the same GitLab account. +- The existing user signs in to GitLab with LDAP for the first time. +- The LDAP email address is the primary email address of an existing GitLab user. If the LDAP email + attribute isn't found in the GitLab user database, a new user is created. -We recommend against using LDAP integration if your LDAP users are -allowed to change their `mail`, `email` or `userPrincipalName` attributes on -the LDAP server, or share email addresses. +If an existing GitLab user wants to enable LDAP sign-in for themselves, they should: -### User deletion +1. Check that their GitLab email address matches their LDAP email address. +1. Sign in to GitLab by using their LDAP credentials. -Users deleted from the LDAP server are immediately blocked from signing in -to GitLab and [no longer consumes a -license](../../../user/admin_area/moderate_users.md). -However, there's an LDAP check cache time of one hour (which is -[configurable](#adjust-ldap-user-sync-schedule) for GitLab Premium users). -This means users already signed-in or who are using Git over SSH can access -GitLab for up to one hour. Manually block the user in the GitLab Admin Area -to immediately block all access. +## Security -## Git password authentication +GitLab has multiple mechanisms to verify a user is still active in LDAP. If the user is no longer active in +LDAP, they are placed in an `ldap_blocked` status and are signed out. They are unable to sign in using any authentication provider until they are +reactivated in LDAP. -LDAP-enabled users can authenticate with Git using their GitLab username or -email and LDAP password, even if password authentication for Git is disabled -in the application settings. +Users are considered inactive in LDAP when they: -## Enable LDAP sign-in for existing GitLab users +- Are removed from the directory completely. +- Reside outside the configured `base` DN or `user_filter` search. +- Are marked as disabled or deactivated in Active Directory through the user account control attribute. This means attribute + `userAccountControl:1.2.840.113556.1.4.803` has bit 2 set. -When a user signs in to GitLab with LDAP for the first time and their LDAP -email address is the primary email address of an existing GitLab user, the -LDAP DN is associated with the existing user. If the LDAP email attribute -isn't found in the GitLab user database, a new user is created. +Status is checked for all LDAP users: -In other words, if an existing GitLab user wants to enable LDAP sign-in for -themselves, they should check that their GitLab email address matches their -LDAP email address, and then sign into GitLab by using their LDAP credentials. +- When signing in using any authentication provider. +- Once per hour for active web sessions or Git requests using tokens or SSH keys. +- When performing Git over HTTP requests using LDAP username and password. +- Once per day during [User Sync](ldap_synchronization.md#user-sync). -## Google Secure LDAP +### Security risks -> Introduced in GitLab 11.9. +You should only use LDAP integration if your LDAP users cannot: -[Google Cloud Identity](https://cloud.google.com/identity/) provides a Secure -LDAP service that can be configured with GitLab for authentication and group sync. -See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instructions. +- Change their `mail`, `email` or `userPrincipalName` attributes on the LDAP server. These + users can potentially take over any account on your GitLab server. +- Share email addresses. LDAP users with the same email address can share the same GitLab + account. ## Configure LDAP @@ -109,7 +103,6 @@ gitlab_rails['ldap_servers'] = { 'verify_certificates' => true, 'bind_dn' => '_the_full_dn_of_the_user_you_will_bind_with', 'password' => '_the_password_of_the_bind_user', - 'verify_certificates' => true, 'tls_options' => { 'ca_file' => '', 'ssl_version' => '', @@ -170,7 +163,7 @@ These configuration settings are available: | `bind_dn` | The full DN of the user you bind with. | **{dotted-circle}** No | `'america\momo'` or `'CN=Gitlab,OU=Users,DC=domain,DC=com'` | | `password` | The password of the bind user. | **{dotted-circle}** No | `'your_great_password'` | | `encryption` | Encryption method. The `method` key is deprecated in favor of `encryption`. | **{check-circle}** Yes | `'start_tls'` or `'simple_tls'` or `'plain'` | -| `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. Defaults to true. | **{dotted-circle}** No | boolean | +| `verify_certificates` | Enables SSL certificate verification if encryption method is `start_tls` or `simple_tls`. If set to false, no validation of the LDAP server's SSL certificate is performed. Defaults to true. | **{dotted-circle}** No | boolean | | `timeout` | Set a timeout, in seconds, for LDAP queries. This helps avoid blocking a request if the LDAP server becomes unresponsive. A value of `0` means there is no timeout. (default: `10`) | **{dotted-circle}** No | `10` or `30` | | `active_directory` | This setting specifies if LDAP server is Active Directory LDAP server. For non-AD servers it skips the AD specific queries. If your LDAP server is not AD, set this to false. | **{dotted-circle}** No | boolean | | `allow_username_or_email_login` | If enabled, GitLab ignores everything after the first `@` in the LDAP username submitted by the user on sign-in. If you are using `uid: 'userPrincipalName'` on ActiveDirectory you must disable this setting, because the userPrincipalName contains an `@`. | **{dotted-circle}** No | boolean | @@ -266,7 +259,7 @@ For more information about `LDAP_MATCHING_RULE_IN_CHAIN` filters, see [Search Filter Syntax](https://docs.microsoft.com/en-us/windows/win32/adsi/search-filter-syntax). Support for nested members in the user filter shouldn't be confused with -[group sync nested groups](#supported-ldap-group-typesattributes) support. +[group sync nested groups](ldap_synchronization.md#supported-ldap-group-typesattributes) support. GitLab does not support the custom filter syntax used by OmniAuth LDAP. @@ -347,7 +340,7 @@ sync, while also allowing your SAML identity provider to handle additional checks like custom 2FA. When LDAP web sign in is disabled, users don't see an **LDAP** tab on the sign-in page. -This does not disable [using LDAP credentials for Git access](#git-password-authentication). +This does not disable using LDAP credentials for Git access. **Omnibus configuration** @@ -458,26 +451,6 @@ If initially your LDAP configuration looked like: 1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. -## Encryption - -### TLS server authentication - -`simple_tls` and `start_tls` are the two available encryption methods. - -For either encryption method, if setting `verify_certificates: false`, TLS -encryption is established with the LDAP server before any LDAP-protocol data is -exchanged but no validation of the LDAP server's SSL certificate is performed. - -### Limitations - -#### TLS client authentication - -Not implemented by `Net::LDAP`. - -You should disable anonymous LDAP authentication and enable simple or Simple Authentication -and Security Layer (SASL) authentication. The TLS client authentication setting in your LDAP server -cannot be mandatory and clients cannot be authenticated with the TLS protocol. - ## Multiple LDAP servers **(PREMIUM SELF)** With GitLab, you can configure multiple LDAP servers that your GitLab instance @@ -528,342 +501,43 @@ If you configure multiple LDAP servers, use a unique naming convention for the `label` section of each entry. That label is used as the display name of the tab shown on the sign-in page. -## User sync **(PREMIUM SELF)** - -Once per day, GitLab runs a worker to check and update GitLab -users against LDAP. - -The process executes the following access checks: - -- Ensure the user is still present in LDAP. -- If the LDAP server is Active Directory, ensure the user is active (not - blocked/disabled state). This check is performed only if - `active_directory: true` is set in the LDAP configuration. - -In Active Directory, a user is marked as disabled/blocked if the user -account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) -has bit 2 set. - -<!-- vale gitlab.Spelling = NO --> - -For more information, see [Bitmask Searches in LDAP](https://ctovswild.com/2009/09/03/bitmask-searches-in-ldap/). - -<!-- vale gitlab.Spelling = YES --> +## Disable anonymous LDAP authentication -The user is set to an `ldap_blocked` state in GitLab if the previous conditions -fail. This means the user cannot sign in or push or pull code. +GitLab doesn't support TLS client authentication. Complete these steps on your LDAP server. -The process also updates the following user information: - -- Email address -- SSH public keys (if `sync_ssh_keys` is set) -- Kerberos identity (if Kerberos is enabled) - -The LDAP sync process: - -- Updates existing users. -- Creates new users on first sign in. - -### Adjust LDAP user sync schedule **(PREMIUM SELF)** - -By default, GitLab runs a worker once per day at 01:30 a.m. server time to -check and update GitLab users against LDAP. - -You can manually configure LDAP user sync times by setting the -following configuration values, in cron format. If needed, you can -use a [crontab generator](http://www.crontabgenerator.com). -The example below shows how to set LDAP user -sync to run once every 12 hours at the top of the hour. - -**Omnibus installations** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" - ``` - -1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -**Source installations** - -1. Edit `config/gitlab.yaml`: - - ```yaml - cron_jobs: - ldap_sync_worker_cron: - "0 */12 * * *" - ``` - -1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. +1. Disable anonymous authentication. +1. Enable one of the following authentication types: + - Simple authentication. + - Simple Authentication and Security Layer (SASL) authentication. -## Group Sync **(PREMIUM SELF)** +The TLS client authentication setting in your LDAP server cannot be mandatory and clients cannot be +authenticated with the TLS protocol. -If your LDAP supports the `memberof` property, when the user signs in for the -first time GitLab triggers a sync for groups the user should be a member of. -That way they don't have to wait for the hourly sync to be granted -access to their groups and projects. +## Deleting users -A group sync process runs every hour on the hour, and `group_base` must be set -in LDAP configuration for LDAP synchronizations based on group CN to work. This allows -GitLab group membership to be automatically updated based on LDAP group members. +Users deleted from the LDAP server: -The `group_base` configuration should be a base LDAP 'container', such as an -'organization' or 'organizational unit', that contains LDAP groups that should -be available to GitLab. For example, `group_base` could be -`ou=groups,dc=example,dc=com`. In the configuration file it looks like the -following. +- Are immediately blocked from signing in to GitLab. +- [No longer consume a license](../../../user/admin_area/moderate_users.md). -**Omnibus configuration** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_servers'] = { - 'main' => { - # snip... - 'group_base' => 'ou=groups,dc=example,dc=com', - } - } - ``` - -1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**Source configuration** - -1. Edit `/home/git/gitlab/config/gitlab.yml`: - - ```yaml - production: - ldap: - servers: - main: - # snip... - group_base: ou=groups,dc=example,dc=com - ``` - -1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. - -To take advantage of group sync, group owners or maintainers must [create one -or more LDAP group links](#add-group-links). - -### Add group links **(PREMIUM SELF)** - -For information on adding group links by using CNs and filters, refer to the -[GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). - -### Administrator sync **(PREMIUM SELF)** - -As an extension of group sync, you can automatically manage your global GitLab -administrators. Specify a group CN for `admin_group` and all members of the -LDAP group are given administrator privileges. The configuration looks -like the following. - -NOTE: -Administrators are not synced unless `group_base` is also -specified alongside `admin_group`. Also, only specify the CN of the `admin_group`, -as opposed to the full DN. -Additionally, if an LDAP user has an `admin` role, but is not a member of the `admin_group` -group, GitLab revokes their `admin` role when syncing. - -**Omnibus configuration** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_servers'] = { - 'main' => { - # snip... - 'group_base' => 'ou=groups,dc=example,dc=com', - 'admin_group' => 'my_admin_group', - } - } - ``` +However, these users can continue to use Git with SSH until the next time the +[LDAP check cache runs](ldap_synchronization.md#adjust-ldap-user-sync-schedule). -1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure). +To delete the account immediately, you can manually +[block the user](../../../user/admin_area/moderate_users.md#block-a-user). -**Source configuration** - -1. Edit `/home/git/gitlab/config/gitlab.yml`: - - ```yaml - production: - ldap: - servers: - main: - # snip... - group_base: ou=groups,dc=example,dc=com - admin_group: my_admin_group - ``` - -1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. - -### Global group memberships lock **(PREMIUM SELF)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) in GitLab 12.0. - -"Lock memberships to LDAP synchronization" setting allows instance administrators -to lock down user abilities to invite new members to a group. - -When enabled, the following applies: - -- Only administrator can manage memberships of any group including access levels. -- Users are not allowed to share project with other groups or invite members to - a project created in a group. - -To enable it, you must: - -1. [Configure LDAP](#configure-ldap). -1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Settings > General**. -1. Expand the **Visibility and access controls** section. -1. Ensure the **Lock memberships to LDAP synchronization** checkbox is selected. - -### Adjust LDAP group sync schedule **(PREMIUM SELF)** - -By default, GitLab runs a group sync process every hour, on the hour. -The values shown are in cron format. If needed, you can use a -[Crontab Generator](http://www.crontabgenerator.com). - -WARNING: -Do not start the sync process too frequently as this -could lead to multiple syncs running concurrently. This concern is primarily -for installations with a large number of LDAP users. Review the -[LDAP group sync benchmark metrics](#benchmarks) to see how -your installation compares before proceeding. - -You can manually configure LDAP group sync times by setting the -following configuration values. The example below shows how to set group -sync to run once every two hours at the top of the hour. - -**Omnibus installations** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" - ``` - -1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. - -**Source installations** - -1. Edit `config/gitlab.yaml`: - - ```yaml - cron_jobs: - ldap_group_sync_worker_cron: - "*/30 * * * *" - ``` - -1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. - -### External groups **(PREMIUM SELF)** - -Using the `external_groups` setting allows you to mark all users belonging -to these groups as [external users](../../../user/permissions.md#external-users). -Group membership is checked periodically through the `LdapGroupSync` background -task. - -**Omnibus configuration** - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab_rails['ldap_servers'] = { - 'main' => { - # snip... - 'external_groups' => ['interns', 'contractors'], - } - } - ``` - -1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure). - -**Source configuration** - -1. Edit `config/gitlab.yaml`: - - ```yaml - production: - ldap: - servers: - main: - # snip... - external_groups: ['interns', 'contractors'] - ``` - -1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. - -### Group sync technical details - -This section outlines what LDAP queries are executed and what behavior you -can expect from group sync. - -Group member access are downgraded from a higher level if their LDAP group -membership changes. For example, if a user the Owner role in a group and the -next group sync reveals they should only have the Developer role, their -access is adjusted accordingly. The only exception is if the user is the -last owner in a group. Groups need at least one owner to fulfill -administrative duties. - -#### Supported LDAP group types/attributes - -GitLab supports LDAP groups that use member attributes: - -- `member` -- `submember` -- `uniquemember` -- `memberof` -- `memberuid` - -This means group sync supports (at least) LDAP groups with the following object -classes: - -- `groupOfNames` -- `posixGroup` -- `groupOfUniqueNames` - -Other object classes should work if members are defined as one of the -mentioned attributes. - -Active Directory supports nested groups. Group sync recursively resolves -membership if `active_directory: true` is set in the configuration file. - -##### Nested group memberships - -Nested group memberships are resolved only if the nested group -is found in the configured `group_base`. For example, if GitLab sees a -nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but -the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` -is ignored. - -#### Queries - -- Each LDAP group is queried a maximum of one time with base `group_base` and - filter `(cn=<cn_from_group_link>)`. -- If the LDAP group has the `memberuid` attribute, GitLab executes another - LDAP query per member to obtain each user's full DN. These queries are - executed with base `base`, scope 'base object', and a filter depending on - whether `user_filter` is set. Filter may be `(uid=<uid_from_group>)` or a - joining of `user_filter`. - -#### Benchmarks +## Google Secure LDAP -Group sync was written to be as performant as possible. Data is cached, database -queries are optimized, and LDAP queries are minimized. The last benchmark run -revealed the following metrics: +> Introduced in GitLab 11.9. -For 20,000 LDAP users, 11,000 LDAP groups, and 1,000 GitLab groups with 10 -LDAP group links each: +[Google Cloud Identity](https://cloud.google.com/identity/) provides a Secure +LDAP service that can be configured with GitLab for authentication and group sync. +See [Google Secure LDAP](google_secure_ldap.md) for detailed configuration instructions. -- Initial sync (no existing members assigned in GitLab) took 1.8 hours -- Subsequent syncs (checking membership, no writes) took 15 minutes +## Synchronize users and groups -These metrics are meant to provide a baseline and performance may vary based on -any number of factors. This benchmark was extreme and most instances don't -have near this many users or groups. Disk speed, database performance, -network and LDAP server response time affects these metrics. +For more information on synchronizing users and groups between LDAP and GitLab, see +[LDAP synchronization](ldap_synchronization.md). ## Troubleshooting diff --git a/doc/administration/auth/ldap/ldap-troubleshooting.md b/doc/administration/auth/ldap/ldap-troubleshooting.md index 4757725d0bd..aa40060c4c1 100644 --- a/doc/administration/auth/ldap/ldap-troubleshooting.md +++ b/doc/administration/auth/ldap/ldap-troubleshooting.md @@ -229,7 +229,7 @@ ldapsearch -H ldaps://$host:$port -D "$bind_dn" -y bind_dn_password.txt -b "$ba #### Sync all users **(PREMIUM SELF)** -The output from a manual [user sync](index.md#user-sync) can show you what happens when +The output from a manual [user sync](ldap_synchronization.md#user-sync) can show you what happens when GitLab tries to sync its users against LDAP. Enter the [rails console](#rails-console) and then run: @@ -239,8 +239,7 @@ Rails.logger.level = Logger::DEBUG LdapSyncWorker.new.perform ``` -Next, [learn how to read the -output](#example-console-output-after-a-user-sync). +Next, [learn how to read the output](#example-console-output-after-a-user-sync). ##### Example console output after a user sync **(PREMIUM SELF)** @@ -342,9 +341,8 @@ LDAP group sync, but for some reason it's not happening. There are several things to check to debug the situation. - Ensure LDAP configuration has a `group_base` specified. - [This configuration](index.md#group-sync) is required for group sync to work properly. -- Ensure the correct [LDAP group link is added to the GitLab - group](index.md#add-group-links). + [This configuration](ldap_synchronization.md#group-sync) is required for group sync to work properly. +- Ensure the correct [LDAP group link is added to the GitLab group](ldap_synchronization.md#add-group-links). - Check that the user has an LDAP identity: 1. Sign in to GitLab as an administrator user. 1. On the top bar, select **Menu > Admin**. @@ -354,7 +352,7 @@ things to check to debug the situation. 1. Select the **Identities** tab. There should be an LDAP identity with an LDAP DN as the 'Identifier'. If not, this user hasn't signed in with LDAP yet and must do so first. -- You've waited an hour or [the configured interval](index.md#adjust-ldap-group-sync-schedule) for +- You've waited an hour or [the configured interval](ldap_synchronization.md#adjust-ldap-group-sync-schedule) for the group to sync. To speed up the process, either go to the GitLab group **Group information > Members** and press **Sync now** (sync one group) or [run the group sync Rake task](../../raketasks/ldap.md#run-a-group-sync) (sync all groups). @@ -366,8 +364,7 @@ the rails console. 1. Choose a GitLab group to test with. This group should have an LDAP group link already configured. 1. [Enable debug logging, find the above GitLab group, and sync it with LDAP](#sync-one-group). -1. Look through the output of the sync. See [example log - output](#example-console-output-after-a-group-sync) +1. Look through the output of the sync. See [example log output](#example-console-output-after-a-group-sync) for how to read the output. 1. If you still aren't able to see why the user isn't being added, [query the LDAP group directly](#query-a-group-in-ldap) to see what members are listed. @@ -377,20 +374,20 @@ the rails console. #### Administrator privileges not granted -When [Administrator sync](index.md#administrator-sync) has been configured +When [Administrator sync](ldap_synchronization.md#administrator-sync) has been configured but the configured users aren't granted the correct administrator privileges, confirm the following are true: -- A [`group_base` is also configured](index.md#group-sync). +- A [`group_base` is also configured](ldap_synchronization.md#group-sync). - The configured `admin_group` in the `gitlab.rb` is a CN, rather than a DN or an array. - This CN falls under the scope of the configured `group_base`. - The members of the `admin_group` have already signed into GitLab with their LDAP credentials. GitLab only grants the Administrator role to the users whose accounts are already connected to LDAP. -If all the above are true and the users are still not getting access, [run a manual -group sync](#sync-all-groups) in the rails console and [look through the -output](#example-console-output-after-a-group-sync) to see what happens when +If all the above are true and the users are still not getting access, +[run a manual group sync](#sync-all-groups) in the rails console and +[look through the output](#example-console-output-after-a-group-sync) to see what happens when GitLab syncs the `admin_group`. #### Sync all groups @@ -399,7 +396,7 @@ NOTE: To sync all groups manually when debugging is unnecessary, [use the Rake task](../../raketasks/ldap.md#run-a-group-sync) instead. -The output from a manual [group sync](index.md#group-sync) can show you what happens +The output from a manual [group sync](ldap_synchronization.md#group-sync) can show you what happens when GitLab syncs its LDAP group memberships against LDAP. ```ruby @@ -494,7 +491,7 @@ this line indicates the sync is finished: Finished syncing admin users for 'ldapmain' provider ``` -If [administrator sync](index.md#administrator-sync) is not configured, you see a message +If [administrator sync](ldap_synchronization.md#administrator-sync) is not configured, you see a message stating as such: ```shell @@ -583,6 +580,25 @@ end You can then [run a UserSync](#sync-all-users) **(PREMIUM SELF)** to sync the latest DN for each of these users. +## Expired license causes errors with multiple LDAP servers + +Using [multiple LDAP servers](index.md#multiple-ldap-servers) requires a valid license. An expired +license can cause: + +- `502` errors in the web interface. +- The following error in logs (the actual strategy name depends on the name configured in `/etc/gitlab/gitlab.rb`): + + ```plaintext + Could not find a strategy with name `Ldapsecondary'. Please ensure it is required or explicitly set it using the :strategy_class option. (Devise::OmniAuth::StrategyNotFound) + ``` + +To resolve this error, you must apply a new license to the GitLab instance without the web interface: + +1. Remove or comment out the GitLab configuration lines for all non-primary LDAP servers. +1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) so that it temporarily uses only one LDAP server. +1. Enter the [Rails console and add the license key](../../troubleshooting/gitlab_rails_cheat_sheet.md#add-a-license-through-the-console). +1. Re-enable the additional LDAP servers in the GitLab configuration and reconfigure GitLab again. + ## Debugging Tools ### LDAP check @@ -610,8 +626,7 @@ If a user account is blocked or unblocked due to the LDAP configuration, a message is [logged to `application.log`](../../logs.md#applicationlog). If there is an unexpected error during an LDAP lookup (configuration error, -timeout), the sign-in is rejected and a message is [logged to -`production.log`](../../logs.md#productionlog). +timeout), the sign-in is rejected and a message is [logged to `production.log`](../../logs.md#productionlog). ### ldapsearch diff --git a/doc/administration/auth/ldap/ldap_synchronization.md b/doc/administration/auth/ldap/ldap_synchronization.md new file mode 100644 index 00000000000..2673a8374ec --- /dev/null +++ b/doc/administration/auth/ldap/ldap_synchronization.md @@ -0,0 +1,349 @@ +--- +stage: Manage +group: Access +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# LDAP synchronization **(PREMIUM SELF)** + +If you have [configured LDAP to work with GitLab](index.md), GitLab can automatically synchronize +users and groups. This process updates user and group information. + +You can change when synchronization occurs. + +## User sync + +Once per day, GitLab runs a worker to check and update GitLab +users against LDAP. + +The process executes the following access checks: + +- Ensure the user is still present in LDAP. +- If the LDAP server is Active Directory, ensure the user is active (not + blocked/disabled state). This check is performed only if + `active_directory: true` is set in the LDAP configuration. + +In Active Directory, a user is marked as disabled/blocked if the user +account control attribute (`userAccountControl:1.2.840.113556.1.4.803`) +has bit 2 set. + +<!-- vale gitlab.Spelling = NO --> + +For more information, see [Bitmask Searches in LDAP](https://ctovswild.com/2009/09/03/bitmask-searches-in-ldap/). + +<!-- vale gitlab.Spelling = YES --> + +The user is set to an `ldap_blocked` state in GitLab if the previous conditions +fail. This means the user cannot sign in or push or pull code. + +The process also updates the following user information: + +- Email address +- SSH public keys (if `sync_ssh_keys` is set) +- Kerberos identity (if Kerberos is enabled) + +The LDAP sync process: + +- Updates existing users. +- Creates new users on first sign in. + +### Adjust LDAP user sync schedule + +By default, GitLab runs a worker once per day at 01:30 a.m. server time to +check and update GitLab users against LDAP. + +You can manually configure LDAP user sync times by setting the +following configuration values, in cron format. If needed, you can +use a [crontab generator](http://www.crontabgenerator.com). +The example below shows how to set LDAP user +sync to run once every 12 hours at the top of the hour. + +**Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_sync_worker_cron'] = "0 */12 * * *" + ``` + +1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source installations** + +1. Edit `config/gitlab.yaml`: + + ```yaml + cron_jobs: + ldap_sync_worker_cron: + "0 */12 * * *" + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +## Group sync + +If your LDAP supports the `memberof` property, when the user signs in for the +first time GitLab triggers a sync for groups the user should be a member of. +That way they don't have to wait for the hourly sync to be granted +access to their groups and projects. + +A group sync process runs every hour on the hour, and `group_base` must be set +in LDAP configuration for LDAP synchronizations based on group CN to work. This allows +GitLab group membership to be automatically updated based on LDAP group members. + +The `group_base` configuration should be a base LDAP 'container', such as an +'organization' or 'organizational unit', that contains LDAP groups that should +be available to GitLab. For example, `group_base` could be +`ou=groups,dc=example,dc=com`. In the configuration file it looks like the +following. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'group_base' => 'ou=groups,dc=example,dc=com', + } + } + ``` + +1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**Source configuration** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +To take advantage of group sync, group owners or maintainers must [create one +or more LDAP group links](#add-group-links). + +### Add group links + +For information on adding group links by using CNs and filters, refer to the +[GitLab groups documentation](../../../user/group/index.md#manage-group-memberships-via-ldap). + +### Administrator sync + +As an extension of group sync, you can automatically manage your global GitLab +administrators. Specify a group CN for `admin_group` and all members of the +LDAP group are given administrator privileges. The configuration looks +like the following. + +NOTE: +Administrators are not synced unless `group_base` is also +specified alongside `admin_group`. Also, only specify the CN of the `admin_group`, +as opposed to the full DN. +Additionally, if an LDAP user has an `admin` role, but is not a member of the `admin_group` +group, GitLab revokes their `admin` role when syncing. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'group_base' => 'ou=groups,dc=example,dc=com', + 'admin_group' => 'my_admin_group', + } + } + ``` + +1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**Source configuration** + +1. Edit `/home/git/gitlab/config/gitlab.yml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + group_base: ou=groups,dc=example,dc=com + admin_group: my_admin_group + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +### Global group memberships lock + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1793) in GitLab 12.0. + +"Lock memberships to LDAP synchronization" setting allows instance administrators +to lock down user abilities to invite new members to a group. + +When enabled, the following applies: + +- Only administrator can manage memberships of any group including access levels. +- Users are not allowed to share project with other groups or invite members to + a project created in a group. + +To enable it, you must: + +1. [Configure LDAP](index.md#configure-ldap). +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > General**. +1. Expand the **Visibility and access controls** section. +1. Ensure the **Lock memberships to LDAP synchronization** checkbox is selected. + +### Adjust LDAP group sync schedule + +By default, GitLab runs a group sync process every hour, on the hour. +The values shown are in cron format. If needed, you can use a +[Crontab Generator](http://www.crontabgenerator.com). + +WARNING: +Do not start the sync process too frequently as this +could lead to multiple syncs running concurrently. This concern is primarily +for installations with a large number of LDAP users. Review the +[LDAP group sync benchmark metrics](#benchmarks) to see how +your installation compares before proceeding. + +You can manually configure LDAP group sync times by setting the +following configuration values. The example below shows how to set group +sync to run once every two hours at the top of the hour. + +**Omnibus installations** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_group_sync_worker_cron'] = "0 */2 * * * *" + ``` + +1. [Reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) for the changes to take effect. + +**Source installations** + +1. Edit `config/gitlab.yaml`: + + ```yaml + cron_jobs: + ldap_group_sync_worker_cron: + "*/30 * * * *" + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +### External groups + +Using the `external_groups` setting allows you to mark all users belonging +to these groups as [external users](../../../user/permissions.md#external-users). +Group membership is checked periodically through the `LdapGroupSync` background +task. + +**Omnibus configuration** + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_rails['ldap_servers'] = { + 'main' => { + # snip... + 'external_groups' => ['interns', 'contractors'], + } + } + ``` + +1. [Apply your changes to GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure). + +**Source configuration** + +1. Edit `config/gitlab.yaml`: + + ```yaml + production: + ldap: + servers: + main: + # snip... + external_groups: ['interns', 'contractors'] + ``` + +1. [Restart GitLab](../../restart_gitlab.md#installations-from-source) for the changes to take effect. + +### Group sync technical details + +This section outlines what LDAP queries are executed and what behavior you +can expect from group sync. + +Group member access are downgraded from a higher level if their LDAP group +membership changes. For example, if a user the Owner role in a group and the +next group sync reveals they should only have the Developer role, their +access is adjusted accordingly. The only exception is if the user is the +last owner in a group. Groups need at least one owner to fulfill +administrative duties. + +#### Supported LDAP group types/attributes + +GitLab supports LDAP groups that use member attributes: + +- `member` +- `submember` +- `uniquemember` +- `memberof` +- `memberuid` + +This means group sync supports (at least) LDAP groups with the following object +classes: + +- `groupOfNames` +- `posixGroup` +- `groupOfUniqueNames` + +Other object classes should work if members are defined as one of the +mentioned attributes. + +Active Directory supports nested groups. Group sync recursively resolves +membership if `active_directory: true` is set in the configuration file. + +##### Nested group memberships + +Nested group memberships are resolved only if the nested group +is found in the configured `group_base`. For example, if GitLab sees a +nested group with DN `cn=nested_group,ou=special_groups,dc=example,dc=com` but +the configured `group_base` is `ou=groups,dc=example,dc=com`, `cn=nested_group` +is ignored. + +#### Queries + +- Each LDAP group is queried a maximum of one time with base `group_base` and + filter `(cn=<cn_from_group_link>)`. +- If the LDAP group has the `memberuid` attribute, GitLab executes another + LDAP query per member to obtain each user's full DN. These queries are + executed with base `base`, scope 'base object', and a filter depending on + whether `user_filter` is set. Filter may be `(uid=<uid_from_group>)` or a + joining of `user_filter`. + +#### Benchmarks + +Group sync was written to be as performant as possible. Data is cached, database +queries are optimized, and LDAP queries are minimized. The last benchmark run +revealed the following metrics: + +For 20,000 LDAP users, 11,000 LDAP groups, and 1,000 GitLab groups with 10 +LDAP group links each: + +- Initial sync (no existing members assigned in GitLab) took 1.8 hours +- Subsequent syncs (checking membership, no writes) took 15 minutes + +These metrics are meant to provide a baseline and performance may vary based on +any number of factors. This benchmark was extreme and most instances don't +have near this many users or groups. Disk speed, database performance, +network and LDAP server response time affects these metrics. diff --git a/doc/administration/auth/oidc.md b/doc/administration/auth/oidc.md index 12729f2050b..b8c443ae4d4 100644 --- a/doc/administration/auth/oidc.md +++ b/doc/administration/auth/oidc.md @@ -27,7 +27,7 @@ The OpenID Connect provides you with a client's details and secret for you to us sudo -u git -H editor config/gitlab.yml ``` - See [Initial OmniAuth Configuration](../../integration/omniauth.md#initial-omniauth-configuration) for initial settings. + See [Configure initial settings](../../integration/omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration. @@ -40,7 +40,7 @@ The OpenID Connect provides you with a client's details and secret for you to us 'icon' => '<custom_provider_icon>', 'args' => { 'name' => 'openid_connect', - 'scope' => ['openid','profile'], + 'scope' => ['openid','profile','email'], 'response_type' => 'code', 'issuer' => '<your_oidc_url>', 'discovery' => true, @@ -65,7 +65,7 @@ The OpenID Connect provides you with a client's details and secret for you to us icon: '<custom_provider_icon>', args: { name: 'openid_connect', - scope: ['openid','profile'], + scope: ['openid','profile','email'], response_type: 'code', issuer: '<your_oidc_url>', discovery: true, @@ -228,7 +228,7 @@ Azure B2C [offers two ways of defining the business logic for logging in a user] While cumbersome to configure, custom policies are required because standard Azure B2C user flows [do not send the OpenID `email` claim](https://github.com/MicrosoftDocs/azure-docs/issues/16566). In -other words, they do not work with the [`allow_single_sign_on` or `auto_link_user` parameters](../../integration/omniauth.md#initial-omniauth-configuration). +other words, they do not work with the [`allow_single_sign_on` or `auto_link_user` parameters](../../integration/omniauth.md#configure-initial-settings). With a standard Azure B2C policy, GitLab cannot create a new account or link to an existing one with an email address. diff --git a/doc/administration/cicd.md b/doc/administration/cicd.md index 89fc31822ee..d53290f1d5d 100644 --- a/doc/administration/cicd.md +++ b/doc/administration/cicd.md @@ -7,22 +7,18 @@ type: howto # GitLab CI/CD instance configuration **(FREE SELF)** -GitLab CI/CD is enabled by default in all new projects on an instance. You can configure -the instance to have [GitLab CI/CD disabled by default](#disable-gitlab-cicd-in-new-projects) -in new projects. - -You can still choose to [enable GitLab CI/CD in individual projects](../ci/enable_or_disable_ci.md#enable-cicd-in-a-project) -at any time. +GitLab administrators can manage the GitLab CI/CD configuration for their instance. ## Disable GitLab CI/CD in new projects -You can set GitLab CI/CD to be disabled by default in all new projects by modifying the settings in: +GitLab CI/CD is enabled by default in all new projects on an instance. You can set +CI/CD to be disabled by default in new projects by modifying the settings in: - `gitlab.yml` for source installations. - `gitlab.rb` for Omnibus GitLab installations. Existing projects that already had CI/CD enabled are unchanged. Also, this setting only changes -the project default, so project owners can still enable CI/CD in the project settings. +the project default, so project owners [can still enable CI/CD in the project settings](../ci/enable_or_disable_ci.md#enable-cicd-in-a-project). For installations from source: @@ -62,6 +58,19 @@ For Omnibus GitLab installations: sudo gitlab-ctl reconfigure ``` +## Set the `needs:` job limit **(FREE SELF)** + +The maximum number of jobs that can be defined in `needs:` defaults to 50. + +A GitLab administrator with [access to the GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session) +can choose a custom limit. For example, to set the limit to `100`: + +```ruby +Plan.default.actual_limits.update!(ci_needs_size_limit: 100) +``` + +To disable directed acyclic graphs (DAG), set the limit to `0`. + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/administration/clusters/kas.md b/doc/administration/clusters/kas.md index 226710a8911..93b24007de8 100644 --- a/doc/administration/clusters/kas.md +++ b/doc/administration/clusters/kas.md @@ -10,7 +10,7 @@ The Kubernetes Agent Server (KAS) is a GitLab backend service dedicated to managing [Kubernetes Agents](../../user/clusters/agent/index.md). The KAS is already installed and available in GitLab.com under `wss://kas.gitlab.com`. -See [how to use GitLab.com's KAS](../../user/clusters/agent/index.md#set-up-the-kubernetes-agent-server). +See [how to use GitLab.com's KAS](../../user/clusters/agent/install/index.md#set-up-the-kubernetes-agent-server). This document describes how to install a KAS for GitLab self-managed instances. ## Installation options diff --git a/doc/administration/compliance.md b/doc/administration/compliance.md index 1761af1ffa1..a05495c024e 100644 --- a/doc/administration/compliance.md +++ b/doc/administration/compliance.md @@ -6,29 +6,81 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Compliance features **(FREE)** -You can configure the following GitLab features to help ensure that your GitLab -instance meets common compliance standards. Click a feature name for additional -documentation. - -The [security features](../security/index.md) in GitLab may also help you meet -relevant compliance standards. - -| Feature | GitLab tier | GitLab SaaS | Product level | -|----------|:-----------:|:-----------:|:-------------:| -|**[Restrict SSH Keys](../security/ssh_keys_restrictions.md)**<br>Control the technology and key length of SSH keys used to access GitLab. | Free+ | **{dotted-circle}** No | Instance | -|**[Granular user roles and flexible permissions](../user/permissions.md)**<br>Manage access and permissions with five different user roles and settings for external users. Set permissions according to people's role, rather than either read or write access to a repository. Don't share the source code with people that only need access to the issue tracker. | Free+ | **{check-circle}** Yes | Instance, Group, Project | -|**[Generate reports on permission levels of users](../user/admin_area/index.md#user-permission-export)**<br>Administrators can generate a report listing all users' access permissions for groups and projects in the instance. | Premium+ | **{dotted-circle}** No | Instance | -|**[Enforce TOS acceptance](../user/admin_area/settings/terms.md)**<br>Enforce your users accepting new terms of service by blocking GitLab traffic. | Free+ | **{dotted-circle}** No | Instance | -|**[Email all users of a project, group, or entire server](../tools/email.md)**<br>An administrator can email groups of users based on project or group membership, or email everyone using the GitLab instance. This is great for scheduled maintenance or upgrades. | Premium+ | **{dotted-circle}** No | Instance | -|**[Omnibus package supports log forwarding](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding)**<br>Forward your logs to a central system. | Premium+ | **{dotted-circle}** No | Instance | -|**[Lock project membership to group](../user/group/index.md#prevent-members-from-being-added-to-a-group)**<br>Group owners can prevent new members from being added to projects within a group. | Premium+ | **{check-circle}** Yes | Group | -|**[LDAP group sync](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition gives administrators the ability to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your product, not configuring your tools. | Premium+ | **{dotted-circle}** No | Instance | -|**[LDAP group sync filters](auth/ldap/index.md#group-sync)**<br>GitLab Enterprise Edition Premium gives more flexibility to synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions. | Premium+ | **{dotted-circle}** No | Instance | -|**[Audit events](audit_events.md)**<br>To maintain the integrity of your code, GitLab Enterprise Edition Premium gives administrators the ability to view any modifications made within the GitLab server in an advanced audit events system, so you can control, analyze, and track every change. | Premium+ | **{check-circle}** Yes | Instance, Group, Project | -|**[Auditor users](auditor_users.md)**<br>Auditor users are users who are given read-only access to all projects, groups, and other resources on the GitLab instance. | Premium+ | **{dotted-circle}** No | Instance | -|**[Credentials inventory](../user/admin_area/credentials_inventory.md)**<br>With a credentials inventory, GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. | Ultimate | **{dotted-circle}** No | Instance | -|**Separation of Duties using [Protected branches](../user/project/protected_branches.md#require-code-owner-approval-on-a-protected-branch) and [custom CI Configuration Paths](../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file)**<br> GitLab Premium users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. View the [Separation of Duties Deploy Project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md) and [Separation of Duties Project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md) to see how to use this set up to define these roles. | Premium+ | **{check-circle}** Yes | Project | -|**[Compliance frameworks](../user/project/settings/index.md#compliance-frameworks)**<br>Create a custom compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. | Premium+ | **{check-circle}** Yes | Group | -|**[Compliance pipelines](../user/project/settings/index.md#compliance-pipeline-configuration)**<br>Define a pipeline configuration to run for any projects with a given compliance framework. | Ultimate | **{check-circle}** Yes | Group | -|**[Compliance report](../user/compliance/compliance_report/index.md)**<br>Quickly get visibility into the compliance posture of your organization. | Ultimate | **{check-circle}** Yes | Group | -|**[External Status Checks](../user/project/merge_requests/status_checks.md)**<br>Interface with third party systems you already use during development to ensure you remain compliant. | Ultimate | **{check-circle}** Yes | Project | +These GitLab features can help ensure that your GitLab instance meets common compliance standards. For more information +about compliance management, see the compliance management [solutions page](https://about.gitlab.com/solutions/compliance/). + +The [security features](../security/index.md) in GitLab may also help you meet relevant compliance standards. + +## Policy management + +Organizations have unique policy requirements, either due to organizational standards or mandates from regulatory bodies. +The following features help you define rules and policies to adhere to workflow requirements, separation of duties, and +secure supply chain best practices: + +- [**Credentials inventory**](../user/admin_area/credentials_inventory.md) (for instances): With a credentials inventory, + GitLab administrators can keep track of the credentials used by all of the users in their GitLab instance. +- [**Granular user roles and flexible permissions**](../user/permissions.md) (for instances, groups, and projects): Manage + access and permissions with five different user roles and settings for external users. Set permissions according to people's + role, rather than either read or write access to a repository. Don't share the source code with people that only need + access to the issue tracker. +- [**Merge request approvals**](../user/project/merge_requests/approvals/index.md) (for instances, groups, and projects): + Configure approvals required for merge requests. +- [**Push rules**](../push_rules/push_rules.md) (for instances, groups, and projects): Control pushes to your + repositories. +- Separation of duties using [**protected branches**](../user/project/protected_branches.md#require-code-owner-approval-on-a-protected-branch) + and [**custom CI/CD configuration paths**](../ci/pipelines/settings.md#specify-a-custom-cicd-configuration-file) (for projects): + Users can leverage the GitLab cross-project YAML configurations to define deployers of code and developers of code. + See how to use this setup to define these roles in: + - The [Separation of Duties deploy project](https://gitlab.com/guided-explorations/separation-of-duties-deploy/blob/master/README.md). + - The [Separation of Duties project](https://gitlab.com/guided-explorations/separation-of-duties/blob/master/README.md). + +## Compliant workflow automation + +It is important for compliance teams to be confident that their controls and requirements are set up correctly, but also that they _stay_ set up correctly. One way of doing this is manually checking settings periodically, but this is error prone and time consuming. A better approach is to use single-source-of-truth settings and automation to ensure that whatever a compliance team has configured, stays configured and working correctly. These features can help you automate compliance: + +- [**Compliance frameworks**](../user/project/settings/index.md#compliance-frameworks) (for groups): Create a custom + compliance framework at the group level to describe the type of compliance requirements any child project needs to follow. +- [**Compliance pipelines**](../user/project/settings/index.md#compliance-pipeline-configuration) (for groups): Define a + pipeline configuration to run for any projects with a given compliance framework. + +## Audit management + +An important part of any compliance program is being able to go back and understand what happened, when +it happened, and who was responsible. This is useful in audit situations as well as for understanding +the root cause of issues when they occur. It is useful to have both low-level, raw lists of audit data +as well as high-level, summary lists of audit data. Between these two, compliance teams can quickly +identify if problems exist and then drill down into the specifics of those issues. These features can help provide visibility into GitLab and audit what is happening: + +- [**Audit events**](audit_events.md) (for instances, groups, and projects): To maintain the integrity of your code, + audit events give administrators the ability to view any modifications made within the GitLab + server in an advanced audit events system, so you can control, analyze, and track every change. +- [**Auditor users**](auditor_users.md) (for instances): Auditor users are users who are given read-only access to all + projects, groups, and other resources on the GitLab instance. +- [**Compliance report**](../user/compliance/compliance_report/index.md) (for groups): Quickly get visibility into the + compliance posture of your organization. + +## Other compliance features + +These features can also help with compliance requirements: + +- [**Email all users of a project, group, or entire server**](../tools/email.md) (for instances): An administrator can + email groups of users based on project or group membership, or email everyone using the GitLab instance. These emails + are great for scheduled maintenance or upgrades. +- [**Enforce ToS acceptance**](../user/admin_area/settings/terms.md) (for instances): Enforce your users accepting new + terms of service by blocking GitLab traffic. +- [**External Status Checks**](../user/project/merge_requests/status_checks.md) (for projects): Interface with third-party + systems you already use during development to ensure you remain compliant. +- [**Generate reports on permission levels of users**](../user/admin_area/index.md#user-permission-export) (for + instances): Administrators can generate a report listing all users' access permissions for groups and projects in the + instance. +- [**Lock project membership to group**](../user/group/index.md#prevent-members-from-being-added-to-a-group) (for + groups): Group owners can prevent new members from being added to projects within a group. +- [**LDAP group sync**](auth/ldap/ldap_synchronization.md#group-sync) (for instances): Gives administrators the ability + to automatically sync groups and manage SSH keys, permissions, and authentication, so you can focus on building your + product, not configuring your tools. +- [**LDAP group sync filters**](auth/ldap/ldap_synchronization.md#group-sync) (for instances): Gives more flexibility to + synchronize with LDAP based on filters, meaning you can leverage LDAP attributes to map GitLab permissions. +- [**Omnibus GitLab package supports log forwarding**](https://docs.gitlab.com/omnibus/settings/logs.html#udp-log-forwarding) + (for instances): Forward your logs to a central system. +- [**Restrict SSH Keys**](../security/ssh_keys_restrictions.md) (for instances): Control the technology and key length + of SSH keys used to access GitLab. diff --git a/doc/administration/environment_variables.md b/doc/administration/environment_variables.md index 3af80363916..21e32d145bd 100644 --- a/doc/administration/environment_variables.md +++ b/doc/administration/environment_variables.md @@ -20,7 +20,7 @@ You can use the following environment variables to override certain values: | Variable | Type | Description | |--------------------------------------------|---------|---------------------------------------------------------------------------------------------------------| | `DATABASE_URL` | string | The database URL; is of the form: `postgresql://localhost/blog_development`. | -| `ENABLE_BOOTSNAP` | string | Enables Bootsnap for speeding up initial Rails boot (`1` to enable). | +| `ENABLE_BOOTSNAP` | string | Toggles [Bootsnap](https://github.com/Shopify/bootsnap) for speeding up initial Rails boot. Enabled by default for non-production environments. Set to `0` to disable. | | `EXTERNAL_URL` | string | Specify the external URL at the [time of installation](https://docs.gitlab.com/omnibus/settings/configuration.html#specifying-the-external-url-at-the-time-of-installation). | | `EXTERNAL_VALIDATION_SERVICE_TIMEOUT` | integer | Timeout, in seconds, for an [external CI/CD pipeline validation service](external_pipeline_validation.md). Default is `5`. | | `EXTERNAL_VALIDATION_SERVICE_URL` | string | URL to an [external CI/CD pipeline validation service](external_pipeline_validation.md). | diff --git a/doc/administration/feature_flags.md b/doc/administration/feature_flags.md index f2067e7a2d1..afbf0759452 100644 --- a/doc/administration/feature_flags.md +++ b/doc/administration/feature_flags.md @@ -42,11 +42,15 @@ GitLab to an earlier version, the feature flag status may change. Features that are disabled by default may change or be removed without notice in a future version of GitLab. -Data corruption, stability degradation, or performance degradation might occur if +Data corruption, stability degradation, performance degradation, or security issues might occur if you enable a feature that's disabled by default. Problems caused by using a default disabled feature aren't covered by GitLab support, unless you were directed by GitLab to enable the feature. +Security issues found in features that are disabled by default are patched in regular releases +and do not follow our regular [maintenance policy](../policy/maintenance.md#security-releases) +with regards to backporting the fix. + ## Risks when disabling released features In most cases, the feature flag code is removed in a future version of GitLab. diff --git a/doc/administration/geo/disaster_recovery/bring_primary_back.md b/doc/administration/geo/disaster_recovery/bring_primary_back.md index d06f11bbe09..64673b9ee8d 100644 --- a/doc/administration/geo/disaster_recovery/bring_primary_back.md +++ b/doc/administration/geo/disaster_recovery/bring_primary_back.md @@ -5,27 +5,27 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto --- -# Bring a demoted primary node back online **(PREMIUM SELF)** +# Bring a demoted primary site back online **(PREMIUM SELF)** -After a failover, it is possible to fail back to the demoted **primary** node to +After a failover, it is possible to fail back to the demoted **primary** site to restore your original configuration. This process consists of two steps: -1. Making the old **primary** node a **secondary** node. -1. Promoting a **secondary** node to a **primary** node. +1. Making the old **primary** site a **secondary** site. +1. Promoting a **secondary** site to a **primary** site. WARNING: -If you have any doubts about the consistency of the data on this node, we recommend setting it up from scratch. +If you have any doubts about the consistency of the data on this site, we recommend setting it up from scratch. -## Configure the former **primary** node to be a **secondary** node +## Configure the former **primary** site to be a **secondary** site -Since the former **primary** node will be out of sync with the current **primary** node, the first step is to bring the former **primary** node up to date. Note, deletion of data stored on disk like -repositories and uploads will not be replayed when bringing the former **primary** node back +Since the former **primary** site will be out of sync with the current **primary** site, the first step is to bring the former **primary** site up to date. Note, deletion of data stored on disk like +repositories and uploads will not be replayed when bringing the former **primary** site back into sync, which may result in increased disk usage. Alternatively, you can [set up a new **secondary** GitLab instance](../setup/index.md) to avoid this. -To bring the former **primary** node up to date: +To bring the former **primary** site up to date: -1. SSH into the former **primary** node that has fallen behind. +1. SSH into the former **primary** site that has fallen behind. 1. Make sure all the services are up: ```shell @@ -33,36 +33,36 @@ To bring the former **primary** node up to date: ``` NOTE: - If you [disabled the **primary** node permanently](index.md#step-2-permanently-disable-the-primary-node), + If you [disabled the **primary** site permanently](index.md#step-2-permanently-disable-the-primary-site), you need to undo those steps now. For Debian/Ubuntu you just need to run `sudo systemctl enable gitlab-runsvdir`. For CentOS 6, you need to install - the GitLab instance from scratch and set it up as a **secondary** node by + the GitLab instance from scratch and set it up as a **secondary** site by following [Setup instructions](../setup/index.md). In this case, you don't need to follow the next step. NOTE: If you [changed the DNS records](index.md#step-4-optional-updating-the-primary-domain-dns-record) - for this node during disaster recovery procedure you may need to [block - all the writes to this node](planned_failover.md#prevent-updates-to-the-primary-node) + for this site during disaster recovery procedure you may need to [block + all the writes to this site](planned_failover.md#prevent-updates-to-the-primary-node) during this procedure. -1. [Set up database replication](../setup/database.md). In this case, the **secondary** node - refers to the former **primary** node. - 1. If [PgBouncer](../../postgresql/pgbouncer.md) was enabled on the **current secondary** node - (when it was a primary node) disable it by editing `/etc/gitlab/gitlab.rb` +1. [Set up database replication](../setup/database.md). In this case, the **secondary** site + refers to the former **primary** site. + 1. If [PgBouncer](../../postgresql/pgbouncer.md) was enabled on the **current secondary** site + (when it was a primary site) disable it by editing `/etc/gitlab/gitlab.rb` and running `sudo gitlab-ctl reconfigure`. - 1. You can then set up database replication on the **secondary** node. + 1. You can then set up database replication on the **secondary** site. -If you have lost your original **primary** node, follow the -[setup instructions](../setup/index.md) to set up a new **secondary** node. +If you have lost your original **primary** site, follow the +[setup instructions](../setup/index.md) to set up a new **secondary** site. -## Promote the **secondary** node to **primary** node +## Promote the **secondary** site to **primary** site -When the initial replication is complete and the **primary** node and **secondary** node are +When the initial replication is complete and the **primary** site and **secondary** site are closely in sync, you can do a [planned failover](planned_failover.md). -## Restore the **secondary** node +## Restore the **secondary** site -If your objective is to have two nodes again, you need to bring your **secondary** -node back online as well by repeating the first step -([configure the former **primary** node to be a **secondary** node](#configure-the-former-primary-node-to-be-a-secondary-node)) -for the **secondary** node. +If your objective is to have two sites again, you need to bring your **secondary** +site back online as well by repeating the first step +([configure the former **primary** site to be a **secondary** site](#configure-the-former-primary-site-to-be-a-secondary-site)) +for the **secondary** site. diff --git a/doc/administration/geo/disaster_recovery/index.md b/doc/administration/geo/disaster_recovery/index.md index f6f88e9b193..bf28eb76ffd 100644 --- a/doc/administration/geo/disaster_recovery/index.md +++ b/doc/administration/geo/disaster_recovery/index.md @@ -16,36 +16,36 @@ For the latest updates, check the [Disaster Recovery epic for complete maturity] Multi-secondary configurations require the complete re-synchronization and re-configuration of all non-promoted secondaries and causes downtime. -## Promoting a **secondary** Geo node in single-secondary configurations +## Promoting a **secondary** Geo site in single-secondary configurations We don't currently provide an automated way to promote a Geo replica and do a failover, but you can do it manually if you have `root` access to the machine. -This process promotes a **secondary** Geo node to a **primary** node. To regain -geographic redundancy as quickly as possible, you should add a new **secondary** node +This process promotes a **secondary** Geo site to a **primary** site. To regain +geographic redundancy as quickly as possible, you should add a new **secondary** site immediately after following these instructions. ### Step 1. Allow replication to finish if possible -If the **secondary** node is still replicating data from the **primary** node, follow +If the **secondary** site is still replicating data from the **primary** site, follow [the planned failover docs](planned_failover.md) as closely as possible in order to avoid unnecessary data loss. -### Step 2. Permanently disable the **primary** node +### Step 2. Permanently disable the **primary** site WARNING: -If the **primary** node goes offline, there may be data saved on the **primary** node -that have not been replicated to the **secondary** node. This data should be treated +If the **primary** site goes offline, there may be data saved on the **primary** site +that have not been replicated to the **secondary** site. This data should be treated as lost if you proceed. -If an outage on the **primary** node happens, you should do everything possible to +If an outage on the **primary** site happens, you should do everything possible to avoid a split-brain situation where writes can occur in two different GitLab instances, complicating recovery efforts. So to prepare for the failover, we -must disable the **primary** node. +must disable the **primary** site. - If you have SSH access: - 1. SSH into the **primary** node to stop and disable GitLab: + 1. SSH into the **primary** site to stop and disable GitLab: ```shell sudo gitlab-ctl stop @@ -57,35 +57,35 @@ must disable the **primary** node. sudo systemctl disable gitlab-runsvdir ``` -- If you do not have SSH access to the **primary** node, take the machine offline and +- If you do not have SSH access to the **primary** site, take the machine offline and prevent it from rebooting by any means at your disposal. You might need to: - Reconfigure the load balancers. - Change DNS records (for example, point the primary DNS record to the - **secondary** node to stop usage of the **primary** node). + **secondary** site to stop usage of the **primary** site). - Stop the virtual servers. - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. + - Revoke object storage permissions from the **primary** site. - Physically disconnect a machine. If you plan to [update the primary domain DNS record](#step-4-optional-updating-the-primary-domain-dns-record), you may wish to lower the TTL now to speed up propagation. -### Step 3. Promoting a **secondary** node +### Step 3. Promoting a **secondary** site WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. +secondary. If the secondary site is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. Note the following when promoting a secondary: -- If replication was paused on the secondary node (for example as a part of +- If replication was paused on the secondary site (for example as a part of upgrading, while you were running a version of GitLab earlier than 13.4), you - _must_ [enable the node by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled) - before proceeding. If the secondary node + _must_ [enable the site by using the database](../replication/troubleshooting.md#message-activerecordrecordinvalid-validation-failed-enabled-geo-primary-node-cannot-be-disabled) + before proceeding. If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), the promotion performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused is lost. @@ -99,7 +99,32 @@ Note the following when promoting a secondary: for more information, see this [troubleshooting advice](../replication/troubleshooting.md#errors-when-using---skip-preflight-checks-or---force). -#### Promoting a **secondary** node running on a single machine +#### Promoting a **secondary** site running on a single node running GitLab 14.5 and later + +1. SSH in to your **secondary** node and execute: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly-promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site running on a single node running GitLab 14.4 and earlier + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. 1. SSH in to your **secondary** node and login as root: @@ -116,7 +141,7 @@ Note the following when promoting a secondary: roles ['geo_secondary_role'] ``` -1. Promote the **secondary** node to the **primary** node: +1. Promote the **secondary** site to the **primary** site: - To promote the secondary node to primary along with [preflight checks](planned_failover.md#preflight-checks): @@ -146,18 +171,57 @@ Note the following when promoting a secondary: gitlab-ctl promote-to-primary-node --force ``` -1. Verify you can connect to the newly-promoted **primary** node using the URL used - previously for the **secondary** node. -1. If successful, the **secondary** node is now promoted to the **primary** node. +1. Verify you can connect to the newly-promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. -#### Promoting a **secondary** node with multiple servers +#### Promoting a **secondary** site with multiple nodes running GitLab 14.5 and later + +1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. SSH into each Rails node on your **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly-promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site with multiple nodes running GitLab 14.4 and earlier + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. The `gitlab-ctl promote-to-primary-node` command cannot be used yet in -conjunction with multiple servers, as it can only -perform changes on a **secondary** with only a single machine. Instead, you must +conjunction with multiple nodes, as it can only perform changes on +a **secondary** with only a single node. Instead, you must do this manually. -1. SSH in to the database node in the **secondary** and trigger PostgreSQL to +1. SSH in to the database node in the **secondary** site and trigger PostgreSQL to promote to read-write: ```shell @@ -187,16 +251,54 @@ do this manually. 1. Verify you can connect to the newly-promoted **primary** using the URL used previously for the **secondary**. -1. If successful, the **secondary** node is now promoted to the **primary** node. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site with a Patroni standby cluster running GitLab 14.5 and later + +1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. SSH into each Rails node on your **secondary** site and run one of the following commands: -#### Promoting a **secondary** node with a Patroni standby cluster + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly-promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site with a Patroni standby cluster running GitLab 14.4 and earlier + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. The `gitlab-ctl promote-to-primary-node` command cannot be used yet in -conjunction with a Patroni standby cluster, as it can only -perform changes on a **secondary** with only a single machine. Instead, you must -do this manually. +conjunction with a Patroni standby cluster, as it can only perform changes on +a **secondary** with only a single node. Instead, you must do this manually. -1. SSH in to the Standby Leader database node in the **secondary** and trigger PostgreSQL to +1. SSH in to the Standby Leader database node in the **secondary** site and trigger PostgreSQL to promote to read-write: ```shell @@ -230,9 +332,81 @@ do this manually. 1. Verify you can connect to the newly-promoted **primary** using the URL used previously for the **secondary**. -1. If successful, the **secondary** node is now promoted to the **primary** node. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site with an external PostgreSQL database running GitLab 14.5 and later + +The `gitlab-ctl geo promote` command can be used in conjunction with +an external PostgreSQL database, but it can only perform changes on +a **secondary** PostgreSQL database managed by Omnibus. +You must promote the replica database associated with the **secondary** +site first. + +1. Promote the replica database associated with the **secondary** site. This + sets the database to read-write. The instructions vary depending on where your database is hosted: + - [Amazon RDS](https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_ReadRepl.html#USER_ReadRepl.Promote) + - [Azure PostgreSQL](https://docs.microsoft.com/en-us/azure/postgresql/howto-read-replicas-portal#stop-replication) + - [Google Cloud SQL](https://cloud.google.com/sql/docs/mysql/replication/manage-replicas#promote-replica) + - For other external PostgreSQL databases, save the following script in your + secondary node, for example `/tmp/geo_promote.sh`, and modify the connection + parameters to match your environment. Then, execute it to promote the replica: + + ```shell + #!/bin/bash -#### Promoting a **secondary** node with an external PostgreSQL database + PG_SUPERUSER=postgres + + # The path to your pg_ctl binary. You may need to adjust this path to match + # your PostgreSQL installation + PG_CTL_BINARY=/usr/lib/postgresql/10/bin/pg_ctl + + # The path to your PostgreSQL data directory. You may need to adjust this + # path to match your PostgreSQL installation. You can also run + # `SHOW data_directory;` from PostgreSQL to find your data directory + PG_DATA_DIRECTORY=/etc/postgresql/10/main + + # Promote the PostgreSQL database and allow read/write operations + sudo -u $PG_SUPERUSER $PG_CTL_BINARY -D $PG_DATA_DIRECTORY promote + ``` + +1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. SSH into each Rails node on your **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly-promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +#### Promoting a **secondary** site with an external PostgreSQL database running GitLab 14.4 and earlier + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. The `gitlab-ctl promote-to-primary-node` command cannot be used in conjunction with an external PostgreSQL database, as it can only perform changes on a **secondary** @@ -287,23 +461,23 @@ required: 1. Verify you can connect to the newly-promoted **primary** using the URL used previously for the **secondary**. -1. If successful, the **secondary** node is now promoted to the **primary** node. +1. If successful, the **secondary** site is now promoted to the **primary** site. ### Step 4. (Optional) Updating the primary domain DNS record -Updating the DNS records for the primary domain to point to the **secondary** node +Updating the DNS records for the primary domain to point to the **secondary** site to prevent the need to update all references to the primary domain to the secondary domain, like changing Git remotes and API URLs. -1. SSH into the **secondary** node and login as root: +1. SSH into the **secondary** site and login as root: ```shell sudo -i ``` 1. Update the primary domain's DNS record. After updating the primary domain's - DNS records to point to the **secondary** node, edit `/etc/gitlab/gitlab.rb` on the - **secondary** node to reflect the new URL: + DNS records to point to the **secondary** site, edit `/etc/gitlab/gitlab.rb` on the + **secondary** site to reflect the new URL: ```ruby # Change the existing external_url configuration @@ -314,13 +488,13 @@ secondary domain, like changing Git remotes and API URLs. Changing `external_url` does not prevent access via the old secondary URL, as long as the secondary DNS records are still intact. -1. Reconfigure the **secondary** node for the change to take effect: +1. Reconfigure the **secondary** site for the change to take effect: ```shell gitlab-ctl reconfigure ``` -1. Execute the command below to update the newly promoted **primary** node URL: +1. Execute the command below to update the newly promoted **primary** site URL: ```shell gitlab-rake geo:update_primary_node_url @@ -335,14 +509,14 @@ secondary domain, like changing Git remotes and API URLs. To determine if you need to do this, search for the `gitlab_rails["geo_node_name"]` setting in your `/etc/gitlab/gitlab.rb` file. If it is commented out with `#` or not found at all, then you - need to update the **primary** node's name in the database. You can search for it + need to update the **primary** site's name in the database. You can search for it like so: ```shell grep "geo_node_name" /etc/gitlab/gitlab.rb ``` - To update the **primary** node's name in the database: + To update the **primary** site's name in the database: ```shell gitlab-rails runner 'Gitlab::Geo.primary_node.update!(name: GeoNode.current_node_name)' @@ -352,12 +526,12 @@ secondary domain, like changing Git remotes and API URLs. If you updated the DNS records for the primary domain, these changes may not have yet propagated depending on the previous DNS records TTL. -### Step 5. (Optional) Add **secondary** Geo node to a promoted **primary** node +### Step 5. (Optional) Add **secondary** Geo site to a promoted **primary** site -Promoting a **secondary** node to **primary** node using the process above does not enable -Geo on the new **primary** node. +Promoting a **secondary** site to **primary** site using the process above does not enable +Geo on the new **primary** site. -To bring a new **secondary** node online, follow the [Geo setup instructions](../index.md#setup-instructions). +To bring a new **secondary** site online, follow the [Geo setup instructions](../index.md#setup-instructions). ### Step 6. (Optional) Removing the secondary's tracking database @@ -376,13 +550,13 @@ for the changes to take effect. ## Promoting secondary Geo replica in multi-secondary configurations -If you have more than one **secondary** node and you need to promote one of them, we suggest you follow -[Promoting a **secondary** Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) +If you have more than one **secondary** site and you need to promote one of them, we suggest you follow +[Promoting a **secondary** Geo site in single-secondary configurations](#promoting-a-secondary-geo-site-in-single-secondary-configurations) and after that you also need two extra steps. -### Step 1. Prepare the new **primary** node to serve one or more **secondary** nodes +### Step 1. Prepare the new **primary** site to serve one or more **secondary** sites -1. SSH into the new **primary** node and login as root: +1. SSH into the new **primary** site and login as root: ```shell sudo -i @@ -442,13 +616,13 @@ and after that you also need two extra steps. ### Step 2. Initiate the replication process -Now we need to make each **secondary** node listen to changes on the new **primary** node. To do that you need +Now we need to make each **secondary** site listen to changes on the new **primary** site. To do that you need to [initiate the replication process](../setup/database.md#step-3-initiate-the-replication-process) again but this time -for another **primary** node. All the old replication settings are overwritten. +for another **primary** site. All the old replication settings are overwritten. ## Promoting a secondary Geo cluster in GitLab Cloud Native Helm Charts -When updating a Cloud Native Geo deployment, the process for updating any node that is external to the secondary Kubernetes cluster does not differ from the non Cloud Native approach. As such, you can always defer to [Promoting a secondary Geo node in single-secondary configurations](#promoting-a-secondary-geo-node-in-single-secondary-configurations) for more information. +When updating a Cloud Native Geo deployment, the process for updating any node that is external to the secondary Kubernetes cluster does not differ from the non Cloud Native approach. As such, you can always defer to [Promoting a secondary Geo site in single-secondary configurations](#promoting-a-secondary-geo-site-in-single-secondary-configurations) for more information. The following sections assume you are using the `gitlab` namespace. If you used a different namespace when setting up your cluster, you should also replace `--namespace gitlab` with your namespace. @@ -489,13 +663,45 @@ must disable the **primary** site: - Revoke object storage permissions from the **primary** site. - Physically disconnect a machine. -### Step 2. Promote all **secondary** nodes external to the cluster +### Step 2. Promote all **secondary** sites external to the cluster WARNING: If the secondary site [has been paused](../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused is lost. +If you are running GitLab 14.5 and later: + +1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. SSH into each Rails node on your **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +If you are running GitLab 14.4 and earlier: + 1. SSH in to the database node in the **secondary** and trigger PostgreSQL to promote to read-write: @@ -522,8 +728,6 @@ Data that was created on the primary while the secondary was paused is lost. After making these changes, [reconfigure GitLab](../../restart_gitlab.md#omnibus-gitlab-reconfigure) on the database node. -### Step 3. Promote the **secondary** cluster - 1. Find the task runner pod: ```shell @@ -536,6 +740,8 @@ Data that was created on the primary while the secondary was paused is lost. kubectl --namespace gitlab exec -ti gitlab-geo-task-runner-XXX -- gitlab-rake geo:set_secondary_as_primary ``` +### Step 3. Promote the **secondary** cluster + 1. Update the existing cluster configuration. You can retrieve the existing configuration with Helm: diff --git a/doc/administration/geo/disaster_recovery/planned_failover.md b/doc/administration/geo/disaster_recovery/planned_failover.md index 6312ed669ae..939b4aec968 100644 --- a/doc/administration/geo/disaster_recovery/planned_failover.md +++ b/doc/administration/geo/disaster_recovery/planned_failover.md @@ -204,4 +204,4 @@ in the loss of any data uploaded to the new **primary** in the meantime. Don't forget to remove the broadcast message after the failover is complete. -Finally, you can bring the [old site back as a secondary](bring_primary_back.md#configure-the-former-primary-node-to-be-a-secondary-node). +Finally, you can bring the [old site back as a secondary](bring_primary_back.md#configure-the-former-primary-site-to-be-a-secondary-site). diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md index 3eb7bc2a8e0..3c7af309f78 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_multi_node.md @@ -66,13 +66,13 @@ promote a Geo replica and perform a failover. NOTE: GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). -On the **secondary** node: +On the **secondary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Geo > Nodes** to see its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of - objects aren't yet replicated (shown in gray), consider giving the node more + objects aren't yet replicated (shown in gray), consider giving the site more time to complete.  @@ -85,20 +85,20 @@ You can use the [Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and the reasons for failure. A common cause of replication failures is the data being missing on the -**primary** node - you can resolve these failures by restoring the data from backup, +**primary** site - you can resolve these failures by restoring the data from backup, or removing references to the missing data. The maintenance window won't end until Geo replication and verification is completely finished. To keep the window as short as possible, you should ensure these processes are close to 100% as possible during active use. -If the **secondary** node is still replicating data from the **primary** node, +If the **secondary** site is still replicating data from the **primary** site, follow these steps to avoid unnecessary data loss: 1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) is implemented, updates must be prevented from happening manually to the - **primary**. Your **secondary** node still needs read-only - access to the **primary** node during the maintenance window: + **primary**. Your **secondary** site still needs read-only + access to the **primary** site during the maintenance window: 1. At the scheduled time, using your cloud provider or your node's firewall, block all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and @@ -121,18 +121,18 @@ follow these steps to avoid unnecessary data loss: ``` From this point, users are unable to view their data or make changes on the - **primary** node. They are also unable to log in to the **secondary** node. + **primary** site. They are also unable to log in to the **secondary** site. However, existing sessions need to work for the remainder of the maintenance period, and so public data is accessible throughout. - 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + 1. Verify the **primary** site is blocked to HTTP traffic by visiting it in browser via another IP. The server should refuse connection. - 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + 1. Verify the **primary** site is blocked to Git over SSH traffic by attempting to pull an existing Git repository with an SSH remote URL. The server should refuse connection. - 1. On the **primary** node: + 1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dhasboard, select **Cron**. @@ -150,7 +150,7 @@ follow these steps to avoid unnecessary data loss: 1. If you are manually replicating any [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. - 1. On the **primary** node: + 1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except @@ -165,7 +165,7 @@ follow these steps to avoid unnecessary data loss: - Database replication lag is 0ms. - The Geo log cursor is up to date (0 events behind). - 1. On the **secondary** node: + 1. On the **secondary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` @@ -173,14 +173,14 @@ follow these steps to avoid unnecessary data loss: 1. [Run an integrity check](../../../raketasks/check.md) to verify the integrity of CI artifacts, LFS objects, and uploads in file storage. - At this point, your **secondary** node contains an up-to-date copy of everything the - **primary** node has, meaning nothing is lost when you fail over. + At this point, your **secondary** site contains an up-to-date copy of everything the + **primary** site has, meaning nothing is lost when you fail over. -1. In this final step, you need to permanently disable the **primary** node. +1. In this final step, you need to permanently disable the **primary** site. WARNING: - When the **primary** node goes offline, there may be data saved on the **primary** node - that has not been replicated to the **secondary** node. This data should be treated + When the **primary** site goes offline, there may be data saved on the **primary** site + that has not been replicated to the **secondary** site. This data should be treated as lost if you proceed. NOTE: @@ -189,9 +189,9 @@ follow these steps to avoid unnecessary data loss: When performing a failover, we want to avoid a split-brain situation where writes can occur in two different GitLab instances. So to prepare for the - failover, you must disable the **primary** node: + failover, you must disable the **primary** site: - - If you have SSH access to the **primary** node, stop and disable GitLab: + - If you have SSH access to the **primary** site, stop and disable GitLab: ```shell sudo gitlab-ctl stop @@ -214,19 +214,58 @@ follow these steps to avoid unnecessary data loss: from starting if the machine reboots as `root` with `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. - - If you do not have SSH access to the **primary** node, take the machine offline and + - If you do not have SSH access to the **primary** site, take the machine offline and prevent it from rebooting. Since there are many ways you may prefer to accomplish this, we avoid a single recommendation. You may need to: - Reconfigure the load balancers. - Change DNS records (for example, point the **primary** DNS record to the - **secondary** node to stop using the **primary** node). + **secondary** site to stop using the **primary** site). - Stop the virtual servers. - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. + - Revoke object storage permissions from the **primary** site. - Physically disconnect a machine. -### Promoting the **secondary** node +### Promoting the **secondary** site running GitLab 14.5 and later + +1. SSH to every Sidekiq, PostgresSQL, and Gitaly node in the **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. SSH into each Rails node on your **secondary** site and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly promoted **primary** site using the URL used + previously for the **secondary** site. +1. If successful, the **secondary** site is now promoted to the **primary** site. + +### Promoting the **secondary** site running GitLab 14.4 and earlier + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. NOTE: A new **secondary** should not be added at this time. If you want to add a new @@ -243,13 +282,13 @@ perform changes on a **secondary** with only a single machine. Instead, you must do this manually. WARNING: -In GitLab 13.2 and 13.3, promoting a secondary node to a primary while the +In GitLab 13.2 and 13.3, promoting a secondary site to a primary while the secondary is paused fails. Do not pause replication before promoting a -secondary. If the node is paused, be sure to resume before promoting. This +secondary. If the site is paused, be sure to resume before promoting. This issue has been fixed in GitLab 13.4 and later. WARNING: - If the secondary node [has been paused](../../../geo/index.md#pausing-and-resuming-replication), this performs +If the secondary site [has been paused](../../../geo/index.md#pausing-and-resuming-replication), this performs a point-in-time recovery to the last known state. Data that was created on the primary while the secondary was paused is lost. @@ -291,6 +330,6 @@ Data that was created on the primary while the secondary was paused is lost. ### Next steps To regain geographic redundancy as quickly as possible, you should -[add a new **secondary** node](../../setup/index.md). To +[add a new **secondary** site](../../setup/index.md). To do that, you can re-add the old **primary** as a new secondary and bring it back online. diff --git a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md index d4782144df8..8a4f2ed4306 100644 --- a/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md +++ b/doc/administration/geo/disaster_recovery/runbooks/planned_failover_single_node.md @@ -54,10 +54,10 @@ promote a Geo replica and perform a failover. NOTE: GitLab 13.9 through GitLab 14.3 are affected by a bug in which the Geo secondary site statuses will appear to stop updating and become unhealthy. For more information, see [Geo Admin Area shows 'Unhealthy' after enabling Maintenance Mode](../../replication/troubleshooting.md#geo-admin-area-shows-unhealthy-after-enabling-maintenance-mode). -On the **secondary** node, navigate to the **Admin Area > Geo** dashboard to +On the **secondary** site, navigate to the **Admin Area > Geo** dashboard to review its status. Replicated objects (shown in green) should be close to 100%, and there should be no failures (shown in red). If a large proportion of -objects aren't yet replicated (shown in gray), consider giving the node more +objects aren't yet replicated (shown in gray), consider giving the site more time to complete.  @@ -70,20 +70,20 @@ You can use the [Geo status API](../../../../api/geo_nodes.md#retrieve-project-sync-or-verification-failures-that-occurred-on-the-current-node) to review failed objects and the reasons for failure. A common cause of replication failures is the data being missing on the -**primary** node - you can resolve these failures by restoring the data from backup, +**primary** site - you can resolve these failures by restoring the data from backup, or removing references to the missing data. The maintenance window won't end until Geo replication and verification is completely finished. To keep the window as short as possible, you should ensure these processes are close to 100% as possible during active use. -If the **secondary** node is still replicating data from the **primary** node, +If the **secondary** site is still replicating data from the **primary** site, follow these steps to avoid unnecessary data loss: 1. Until a [read-only mode](https://gitlab.com/gitlab-org/gitlab/-/issues/14609) is implemented, updates must be prevented from happening manually to the - **primary**. Your **secondary** node still needs read-only - access to the **primary** node during the maintenance window: + **primary**. Your **secondary** site still needs read-only + access to the **primary** site during the maintenance window: 1. At the scheduled time, using your cloud provider or your node's firewall, block all HTTP, HTTPS and SSH traffic to/from the **primary** node, **except** for your IP and @@ -106,18 +106,18 @@ follow these steps to avoid unnecessary data loss: ``` From this point, users are unable to view their data or make changes on the - **primary** node. They are also unable to log in to the **secondary** node. + **primary** site. They are also unable to log in to the **secondary** site. However, existing sessions need to work for the remainder of the maintenance period, and so public data is accessible throughout. - 1. Verify the **primary** node is blocked to HTTP traffic by visiting it in browser via + 1. Verify the **primary** site is blocked to HTTP traffic by visiting it in browser via another IP. The server should refuse connection. - 1. Verify the **primary** node is blocked to Git over SSH traffic by attempting to pull an + 1. Verify the **primary** site is blocked to Git over SSH traffic by attempting to pull an existing Git repository with an SSH remote URL. The server should refuse connection. - 1. On the **primary** node: + 1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dhasboard, select **Cron**. @@ -135,7 +135,7 @@ follow these steps to avoid unnecessary data loss: 1. If you are manually replicating any [data not managed by Geo](../../replication/datatypes.md#limitations-on-replicationverification), trigger the final replication process now. - 1. On the **primary** node: + 1. On the **primary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all queues except @@ -143,14 +143,14 @@ follow these steps to avoid unnecessary data loss: These queues contain work that has been submitted by your users; failing over before it is completed, causes the work to be lost. 1. On the left sidebar, select **Geo > Nodes** and wait for the - following conditions to be true of the **secondary** node you are failing over to: + following conditions to be true of the **secondary** site you are failing over to: - All replication meters reach 100% replicated, 0% failures. - All verification meters reach 100% verified, 0% failures. - Database replication lag is 0ms. - The Geo log cursor is up to date (0 events behind). - 1. On the **secondary** node: + 1. On the **secondary** site: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Jobs**. 1. On the Sidekiq dashboard, select **Queues**, and wait for all the `geo` @@ -158,14 +158,14 @@ follow these steps to avoid unnecessary data loss: 1. [Run an integrity check](../../../raketasks/check.md) to verify the integrity of CI artifacts, LFS objects, and uploads in file storage. - At this point, your **secondary** node contains an up-to-date copy of everything the - **primary** node has, meaning nothing is lost when you fail over. + At this point, your **secondary** site contains an up-to-date copy of everything the + **primary** site has, meaning nothing is lost when you fail over. -1. In this final step, you need to permanently disable the **primary** node. +1. In this final step, you need to permanently disable the **primary** site. WARNING: - When the **primary** node goes offline, there may be data saved on the **primary** node - that has not been replicated to the **secondary** node. This data should be treated + When the **primary** site goes offline, there may be data saved on the **primary** site + that has not been replicated to the **secondary** site. This data should be treated as lost if you proceed. NOTE: @@ -174,9 +174,9 @@ follow these steps to avoid unnecessary data loss: When performing a failover, we want to avoid a split-brain situation where writes can occur in two different GitLab instances. So to prepare for the - failover, you must disable the **primary** node: + failover, you must disable the **primary** site: - - If you have SSH access to the **primary** node, stop and disable GitLab: + - If you have SSH access to the **primary** site, stop and disable GitLab: ```shell sudo gitlab-ctl stop @@ -199,19 +199,19 @@ follow these steps to avoid unnecessary data loss: from starting if the machine reboots as `root` with `initctl stop gitlab-runsvvdir && echo 'manual' > /etc/init/gitlab-runsvdir.override && initctl reload-configuration`. - - If you do not have SSH access to the **primary** node, take the machine offline and + - If you do not have SSH access to the **primary** site, take the machine offline and prevent it from rebooting. Since there are many ways you may prefer to accomplish this, we avoid a single recommendation. You may need to: - Reconfigure the load balancers. - Change DNS records (for example, point the **primary** DNS record to the - **secondary** node to stop using the **primary** node). + **secondary** site to stop using the **primary** site). - Stop the virtual servers. - Block traffic through a firewall. - - Revoke object storage permissions from the **primary** node. + - Revoke object storage permissions from the **primary** site. - Physically disconnect a machine. -### Promoting the **secondary** node +### Promoting the **secondary** site Note the following when promoting a secondary: @@ -222,9 +222,35 @@ Note the following when promoting a secondary: error during this process, read [the troubleshooting advice](../../replication/troubleshooting.md#fixing-errors-during-a-failover-or-when-promoting-a-secondary-to-a-primary-node). -To promote the secondary node: +To promote the secondary site running GitLab 14.5 and later: -1. SSH in to your **secondary** node and login as root: +1. SSH in to your **secondary** node and run one of the following commands: + + - To promote the secondary node to primary: + + ```shell + sudo gitlab-ctl geo promote + ``` + + - To promote the secondary node to primary **without any further confirmation**: + + ```shell + sudo gitlab-ctl geo promote --force + ``` + +1. Verify you can connect to the newly promoted **primary** site using the URL used + previously for the **secondary** site. + + If successful, the **secondary** site is now promoted to the **primary** site. + +To promote the secondary site running GitLab 14.4 and earlier: + +WARNING: +The `gitlab-ctl promote-to-primary-node` and `gitlab-ctl promoted-db` commands are +deprecated in GitLab 14.5 and later, and are scheduled to [be removed in GitLab 15.0](https://gitlab.com/gitlab-org/gitlab/-/issues/345207). +Use `gitlab-ctl geo promote` instead. + +1. SSH in to your **secondary** site and login as root: ```shell sudo -i @@ -275,20 +301,20 @@ To promote the secondary node: gitlab-ctl promote-to-primary-node --skip-preflight-check ``` - You can also promote the secondary node to primary **without any further confirmation**, even when preflight checks fail: + You can also promote the secondary site to primary **without any further confirmation**, even when preflight checks fail: ```shell sudo gitlab-ctl promote-to-primary-node --force ``` -1. Verify you can connect to the newly promoted **primary** node using the URL used - previously for the **secondary** node. +1. Verify you can connect to the newly promoted **primary** site using the URL used + previously for the **secondary** site. - If successful, the **secondary** node has now been promoted to the **primary** node. + If successful, the **secondary** site is now promoted to the **primary** site. ### Next steps To regain geographic redundancy as quickly as possible, you should -[add a new **secondary** node](../../setup/index.md). To +[add a new **secondary** site](../../setup/index.md). To do that, you can re-add the old **primary** as a new secondary and bring it back online. diff --git a/doc/administration/geo/index.md b/doc/administration/geo/index.md index 48091967189..30d8d765dc5 100644 --- a/doc/administration/geo/index.md +++ b/doc/administration/geo/index.md @@ -129,25 +129,38 @@ and we recommend you use: ### Firewall rules -The following table lists basic ports that must be open between the **primary** and **secondary** sites for Geo. +The following table lists basic ports that must be open between the **primary** and **secondary** sites for Geo. To simplify failovers, we recommend opening ports in both directions. -| **Primary** site | **Secondary** site | Protocol | -|:-----------------|:-------------------|:-------------| -| 80 | 80 | HTTP | -| 443 | 443 | TCP or HTTPS | -| 22 | 22 | TCP | -| 5432 | | PostgreSQL | +| Source site | Source port | Destination site | Destination port | Protocol | +|-------------|-------------|------------------|------------------|-------------| +| Primary | Any | Secondary | 80 | TCP (HTTP) | +| Primary | Any | Secondary | 443 | TCP (HTTPS) | +| Secondary | Any | Primary | 80 | TCP (HTTP) | +| Secondary | Any | Primary | 443 | TCP (HTTPS) | +| Secondary | Any | Primary | 5432 | TCP | See the full list of ports used by GitLab in [Package defaults](../package_information/defaults.md) NOTE: -[Web terminal](../../ci/environments/index.md#web-terminals) support requires your load balancer to correctly handle WebSocket connections. +[Web terminal](../../ci/environments/index.md#web-terminals-deprecated) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the [web terminal](../integration/terminal.md) integration guide for more details. NOTE: When using HTTPS protocol for port 443, you need to add an SSL certificate to the load balancers. If you wish to terminate SSL at the GitLab application server instead, use TCP protocol. +#### Internal URL + +HTTP requests from any Geo secondary site to the primary Geo site use the Internal URL of the primary +Geo site. If this is not explicitly defined in the primary Geo site settings in the Admin Area, the +public URL of the primary site will be used. + +To update the internal URL of the primary Geo site: + +1. On the top bar, go to **Menu > Admin > Geo > Sites**. +1. Select **Edit** on the primary site. +1. Change the **Internal URL**, then select **Save changes**. + ### LDAP We recommend that if you use LDAP on your **primary** site, you also set up secondary LDAP servers on each **secondary** site. Otherwise, users are unable to perform Git operations over HTTP(s) on the **secondary** site using HTTP Basic Authentication. However, Git via SSH and personal access tokens still works. @@ -258,6 +271,10 @@ For information on using Geo in disaster recovery situations to mitigate data-lo For more information on how to replicate the Container Registry, see [Docker Registry for a **secondary** site](replication/docker_registry.md). +### Geo secondary proxy + +For more information on using Geo proxying on secondary nodes, see [Geo proxying for secondary sites](secondary_proxy/index.md). + ### Security Review For more information on Geo security, see [Geo security review](replication/security_review.md). diff --git a/doc/administration/geo/replication/datatypes.md b/doc/administration/geo/replication/datatypes.md index e8e87f92481..c98436157fc 100644 --- a/doc/administration/geo/replication/datatypes.md +++ b/doc/administration/geo/replication/datatypes.md @@ -183,14 +183,13 @@ successfully, you must replicate their data using some other means. |[Application data in PostgreSQL](../../postgresql/index.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[Project repository](../../../user/project/repository/) | **Yes** (10.2) | **Yes** (10.7) | No | | |[Project wiki repository](../../../user/project/wiki/) | **Yes** (10.2) | **Yes** (10.7) | No | | -|[Group wiki repository](../../../user/project/wiki/index.md#group-wikis) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | +|[Group wiki repository](../../../user/project/wiki/group.md) | [**Yes** (13.10)](https://gitlab.com/gitlab-org/gitlab/-/issues/208147) | No | No | Behind feature flag `geo_group_wiki_repository_replication`, enabled by default. | |[Uploads](../../uploads.md) | **Yes** (10.2) | [No](https://gitlab.com/groups/gitlab-org/-/epics/1817) | No | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | |[LFS objects](../../lfs/index.md) | **Yes** (10.2) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8922) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. GitLab versions 11.11.x and 12.0.x are affected by [a bug that prevents any new LFS objects from replicating](https://gitlab.com/gitlab-org/gitlab/-/issues/32696).<br /><br />Behind feature flag `geo_lfs_object_replication`, enabled by default. | |[Personal snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | |[Project snippets](../../../user/snippets.md) | **Yes** (10.2) | **Yes** (10.2) | No | | -|[CI job artifacts (other than Job Logs)](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | +|[CI job artifacts](../../../ci/pipelines/job_artifacts.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. Job logs also verified on transfer. | |[CI Pipeline Artifacts](https://gitlab.com/gitlab-org/gitlab/-/blob/master/app/models/ci/pipeline_artifact.rb) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | [**Yes** (13.11)](https://gitlab.com/gitlab-org/gitlab/-/issues/238464) | Via Object Storage provider if supported. Native Geo support (Beta). | Persists additional artifacts after a pipeline completes | -|[Job logs](../../job_logs.md) | **Yes** (10.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/8923) | Via Object Storage provider if supported. Native Geo support (Beta). | Verified only on transfer or manually using [Integrity Check Rake Task](../../raketasks/check.md) on both sites and comparing the output between them. | |[Container Registry](../../packages/container_registry.md) | **Yes** (12.3) | No | No | Disabled by default. See [instructions](docker_registry.md) to enable. | |[Content in object storage (beta)](object_storage.md) | **Yes** (12.4) | [No](https://gitlab.com/gitlab-org/gitlab/-/issues/13845) | No | | |[Infrastructure Registry](../../../user/packages/infrastructure_registry/index.md) | **Yes** (14.0) | [**Yes**](#limitation-of-verification-for-files-in-object-storage) (14.0) | Via Object Storage provider if supported. Native Geo support (Beta). | Behind feature flag `geo_package_file_replication`, enabled by default. | diff --git a/doc/administration/geo/replication/disable_geo.md b/doc/administration/geo/replication/disable_geo.md index ae2597ad649..02a65f0b8e1 100644 --- a/doc/administration/geo/replication/disable_geo.md +++ b/doc/administration/geo/replication/disable_geo.md @@ -28,7 +28,7 @@ To disable Geo, you need to first remove all your secondary Geo sites, which mea anymore on these sites. You can follow our docs to [remove your secondary Geo sites](remove_geo_site.md). If the current site that you want to keep using is a secondary site, you need to first promote it to primary. -You can use our steps on [how to promote a secondary site](../disaster_recovery/#step-3-promoting-a-secondary-node) +You can use our steps on [how to promote a secondary site](../disaster_recovery/#step-3-promoting-a-secondary-site) to do that. ## Remove the primary site from the UI diff --git a/doc/administration/geo/replication/troubleshooting.md b/doc/administration/geo/replication/troubleshooting.md index b7370d32056..432d042608c 100644 --- a/doc/administration/geo/replication/troubleshooting.md +++ b/doc/administration/geo/replication/troubleshooting.md @@ -455,7 +455,7 @@ To solve this: 1. Back up [the `.git` folder](../../repository_storage_types.md#translate-hashed-storage-paths). -1. Optional: [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem)) +1. Optional: [Spot-check](../../troubleshooting/log_parsing.md#find-all-projects-affected-by-a-fatal-git-problem) a few of those IDs whether they indeed correspond to a project with known Geo replication failures. Use `fatal: 'geo'` as the `grep` term and the following API call: @@ -683,7 +683,7 @@ when promoting a secondary to a primary node with strategies to resolve them. ### Message: ActiveRecord::RecordInvalid: Validation failed: Name has already been taken -When [promoting a **secondary** node](../disaster_recovery/index.md#step-3-promoting-a-secondary-node), +When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), you might encounter the following error: ```plaintext @@ -723,21 +723,35 @@ If you disabled a secondary node, either with the [replication pause task](../in (13.2) or by using the user interface (13.1 and earlier), you must first re-enable the node before you can continue. This is fixed in 13.4. -Run the following command, replacing `https://<secondary url>/` with the URL -for your secondary server, using either `http` or `https`, and ensuring that you -end the URL with a slash (`/`): +This can be fixed in the database. -```shell -sudo gitlab-rails dbconsole +1. Start a database console: -UPDATE geo_nodes SET enabled = true WHERE url = 'https://<secondary url>/' AND enabled = false;" -``` + In [GitLab 14.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/341210): + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + In GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + +1. Run the following command, replacing `https://<secondary url>/` with the URL + for your secondary server. You can use either `http` or `https`, but ensure that you + end the URL with a slash (`/`): + + ```sql + UPDATE geo_nodes SET enabled = true WHERE url = 'https://<secondary url>/' AND enabled = false;" + ``` -This should update 1 row. + This should update 1 row. ### Message: ``NoMethodError: undefined method `secondary?' for nil:NilClass`` -When [promoting a **secondary** node](../disaster_recovery/index.md#step-3-promoting-a-secondary-node), +When [promoting a **secondary** site](../disaster_recovery/index.md#step-3-promoting-a-secondary-site), you might encounter the following error: ```plaintext @@ -753,13 +767,13 @@ Tasks: TOP => geo:set_secondary_as_primary (See full trace by running task with --trace) ``` -This command is intended to be executed on a secondary node only, and this error -is displayed if you attempt to run this command on a primary node. +This command is intended to be executed on a secondary site only, and this error +is displayed if you attempt to run this command on a primary site. ### Message: `sudo: gitlab-pg-ctl: command not found` When -[promoting a **secondary** node with multiple servers](../disaster_recovery/index.md#promoting-a-secondary-node-with-multiple-servers), +[promoting a **secondary** site with multiple nodes](../disaster_recovery/index.md#promoting-a-secondary-site-with-multiple-nodes-running-gitlab-144-and-earlier), you need to run the `gitlab-pg-ctl` command to promote the PostgreSQL read-replica database. diff --git a/doc/administration/geo/secondary_proxy/img/single_url_add_traffic_policy_endpoints.png b/doc/administration/geo/secondary_proxy/img/single_url_add_traffic_policy_endpoints.png Binary files differnew file mode 100644 index 00000000000..d1c4afceb04 --- /dev/null +++ b/doc/administration/geo/secondary_proxy/img/single_url_add_traffic_policy_endpoints.png diff --git a/doc/administration/geo/secondary_proxy/img/single_url_create_policy_records_with_traffic_policy.png b/doc/administration/geo/secondary_proxy/img/single_url_create_policy_records_with_traffic_policy.png Binary files differnew file mode 100644 index 00000000000..3bad391f220 --- /dev/null +++ b/doc/administration/geo/secondary_proxy/img/single_url_create_policy_records_with_traffic_policy.png diff --git a/doc/administration/geo/secondary_proxy/index.md b/doc/administration/geo/secondary_proxy/index.md new file mode 100644 index 00000000000..2b8c0d1e6fa --- /dev/null +++ b/doc/administration/geo/secondary_proxy/index.md @@ -0,0 +1,127 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Geo proxying for secondary sites **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5914) in GitLab 14.4 [with a flag](../../feature_flags.md) named `geo_secondary_proxy`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. See below to [Set up a unified URL for Geo sites](#set-up-a-unified-url-for-geo-sites). +The feature is not ready for production use. + +Use Geo proxying to: + +- Have secondary sites serve read-write traffic by proxying to the primary site. +- Selectively accelerate replicated data types by directing read-only operations to the local site instead. + +When enabled, users of the secondary site can use the WebUI as if they were accessing the +primary site's UI. This significantly improves the overall user experience of secondary sites. + +With secondary proxying, web requests to secondary Geo sites are +proxied directly to the primary, and appear to act as a read-write site. + +Proxying is done by the [`gitlab-workhorse`](https://gitlab.com/gitlab-org/gitlab-workhorse) component. +Traffic usually sent to the Rails application on the Geo secondary site is proxied +to the [internal URL](../index.md#internal-url) of the primary Geo site instead. + +Use secondary proxying for use-cases including: + +- Having all Geo sites behind a single URL. +- Geographically load-balancing traffic without worrying about write access. + +## Set up a unified URL for Geo sites + +Secondary sites can transparently serve read-write traffic. You can +use a single external URL so that requests can hit either the primary Geo site +or any secondary Geo sites that use Geo proxying. + +### Configure an external URL to send traffic to both sites + +Follow the [Location-aware public URL](location_aware_external_url.md) steps to create +a single URL used by all Geo sites, including the primary. + +### Update the Geo sites to use the same external URL + +1. On your Geo sites, SSH **into each node running Rails (Puma, Sidekiq, Log-Cursor) + and change the `external_url` to that of the single URL: + + ```shell + sudo editor /etc/gitlab/gitlab.rb + ``` + +1. Reconfigure the updated nodes for the change to take effect if the URL was different than the one already set: + + ```shell + gitlab-ctl reconfigure + ``` + +1. To match the new external URL set on the secondary Geo sites, the primary database + needs to reflect this change. + + In the Geo administration page of the **primary** site, edit each Geo secondary that + is using the secondary proxying and set the `URL` field to the single URL. + Make sure the primary site is also using this URL. + +### Enable secondary proxying + +1. SSH into each application node (serving user traffic directly) on your secondary Geo site + and add the following environment variable: + + ```shell + sudo editor /etc/gitlab/gitlab.rb + ``` + + ```ruby + gitlab_workhorse['env'] = { + "GEO_SECONDARY_PROXY" => "1" + } + ``` + +1. Reconfigure the updated nodes for the change to take effect: + + ```shell + gitlab-ctl reconfigure + ``` + +1. SSH into one node running Rails on your primary Geo site and enable the Geo secondary proxy feature flag: + + ```shell + sudo gitlab-rails runner "Feature.enable(:geo_secondary_proxy)" + ``` + +## Enable Geo proxying with Separate URLs + +The ability to use proxying with separate URLs is still in development. You can follow the +["Geo secondary proxying with separate URLs" epic](https://gitlab.com/groups/gitlab-org/-/epics/6865) +for progress. + +## Features accelerated by secondary Geo sites + +Most HTTP traffic sent to a secondary Geo site can be proxied to the primary Geo site. With this architecture, +secondary Geo sites are able to support write requests. Certain **read** requests are handled locally by secondary +sites for improved latency and bandwidth nearby. All write requests are proxied to the primary site. + +The following table details the components currently tested through the Geo secondary site Workhorse proxy. +It does not cover all data types, more will be added in the future as they are tested. + +| Feature / component | Accelerated reads? | +|:----------------------------------------------------|:-----------------------| +| Project, wiki, design repository (using the web UI) | **{dotted-circle}** No | +| Project, wiki repository (using Git) | **{check-circle}** Yes <sup>1</sup> | +| Project, Personal Snippet (using the web UI) | **{dotted-circle}** No | +| Project, Personal Snippet (using Git) | **{check-circle}** Yes <sup>1</sup> | +| Group wiki repository (using the web UI) | **{dotted-circle}** No | +| Group wiki repository (using Git) | **{check-circle}** Yes <sup>1</sup> | +| User uploads | **{dotted-circle}** No | +| LFS objects (using the web UI) | **{dotted-circle}** No | +| LFS objects (using Git) | **{check-circle}** Yes | +| Pages | **{dotted-circle}** No <sup>2</sup> | +| Advanced search (using the web UI) | **{dotted-circle}** No | + +1. Git reads are served from the local secondary while pushes get proxied to the primary. + Selective sync or cases where repositories don't exist locally on the Geo secondary throw a "not found" error. +1. Pages can use the same URL (without access control), but must be configured separately and are not proxied. diff --git a/doc/administration/geo/secondary_proxy/location_aware_external_url.md b/doc/administration/geo/secondary_proxy/location_aware_external_url.md new file mode 100644 index 00000000000..b5a8d4baa1f --- /dev/null +++ b/doc/administration/geo/secondary_proxy/location_aware_external_url.md @@ -0,0 +1,83 @@ +--- +stage: Enablement +group: Geo +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: howto +--- + +# Location-aware public URL **(PREMIUM SELF)** + +With [Geo proxying for secondary sites](index.md), you can provide GitLab users +with a single URL that automatically uses the Geo site closest to them. +Users don't need to use different URLs or worry about read-only operations to take +advantage of closer Geo sites as they move. + +With [Geo proxying for secondary sites](index.md) web and Git requests are proxied +from **secondary** sites to the **primary**. + +Though these instructions use [AWS Route53](https://aws.amazon.com/route53/), +other services such as [Cloudflare](https://www.cloudflare.com/) can be used +as well. + +## Prerequisites + +This example creates a `gitlab.example.com` subdomain that automatically directs +requests: + +- From Europe to a **secondary** site. +- From all other locations to the **primary** site. + +The URLs to access each node by itself are: + +- `primary.example.com` as a Geo **primary** site. +- `secondary.example.com` as a Geo **secondary** site. + +For this example, you need: + +- A working GitLab **primary** site that is accessible at `gitlab.example.com` _and_ `primary.example.com`. +- A working GitLab **secondary** site. +- A Route53 Hosted Zone managing your domain for the Route53 setup. + +If you haven't yet set up a Geo _primary_ site and _secondary_ site, see the +[Geo setup instructions](../index.md#setup-instructions). + +## AWS Route53 + +### Create a traffic policy + +In a Route53 Hosted Zone, traffic policies can be used to set up a variety of +routing configurations. + +1. Go to the +[Route53 dashboard](https://console.aws.amazon.com/route53/home) and select +**Traffic policies**. + +1. Select **Create traffic policy**. +1. Fill in the **Policy Name** field with `Single Git Host` and select **Next**. +1. Leave **DNS type** as `A: IP Address in IPv4 format`. +1. Select **Connect to...**, then **Geolocation rule**. +1. For the first **Location**: + 1. Leave it as `Default`. + 1. Select **Connect to...**, then **New endpoint**. + 1. Choose **Type** `value` and fill it in with `<your **primary** IP address>`. + +1. For the second **Location**: + 1. Choose `Europe`. + 1. Select **Connect to...** and select **New endpoint**. + 1. Choose **Type** `value` and fill it in with `<your **secondary** IP address>`. + +  + +1. Select **Create traffic policy**. +1. Fill in **Policy record DNS name** with `gitlab`. + +  + +1. Select **Create policy records**. + +You have successfully set up a single host, like `gitlab.example.com`, which +distributes traffic to your Geo sites by geolocation. + +## Enable Geo proxying for secondary sites + +After setting up a single URL to use for all Geo sites, continue with the [steps to enable Geo proxying for secondary sites](index.md). diff --git a/doc/administration/geo/setup/database.md b/doc/administration/geo/setup/database.md index d72bb0737ae..7f4c771c962 100644 --- a/doc/administration/geo/setup/database.md +++ b/doc/administration/geo/setup/database.md @@ -7,23 +7,21 @@ type: howto # Geo database replication **(PREMIUM SELF)** -NOTE: -If your GitLab installation uses external (not managed by Omnibus) PostgreSQL -instances, the Omnibus roles are unable to perform all necessary -configuration steps. In this case, -[follow the Geo with external PostgreSQL instances document instead](external_database.md). +This document describes the minimal required steps to replicate your primary +GitLab database to a secondary node's database. You may have to change some +values, based on attributes including your database's setup and size. NOTE: -The stages of the setup process must be completed in the documented order. -Before attempting the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). +If your GitLab installation uses external (not managed by Omnibus GitLab) +PostgreSQL instances, the Omnibus roles cannot perform all necessary +configuration steps. In this case, use the [Geo with external PostgreSQL instances](external_database.md) +process instead. -This document describes the minimal steps you have to take to replicate your -**primary** GitLab database to a **secondary** node's database. You may have to -change some values, based on attributes including your database's setup and -size. +The stages of the setup process must be completed in the documented order. +Before you attempt the steps in this stage, [complete all prior stages](../setup/index.md#using-omnibus-gitlab). -You are encouraged to first read through all the steps before executing them -in your testing/production environment. +Be sure to read and review all of these steps before you execute them in your +testing or production environments. ## Single instance database replication @@ -214,7 +212,7 @@ There is an [issue where support is being discussed](https://gitlab.com/gitlab-o ## - Prevents automatic upgrade of Postgres since it requires downtime of ## streaming replication to Geo secondary sites ## - Enables standard single-node GitLab services like NGINX, Puma, Redis, - ## Sidekiq, etc. If you are segregating services, then you will need to + ## or Sidekiq. If you are segregating services, then you will need to ## explicitly disable unwanted services. ## roles(['geo_primary_role']) @@ -690,8 +688,8 @@ could do it with [HAProxy](https://www.haproxy.org/). The following IPs and names are used as an example: - `10.6.0.21`: Patroni 1 (`patroni1.internal`) -- `10.6.0.21`: Patroni 2 (`patroni2.internal`) -- `10.6.0.22`: Patroni 3 (`patroni3.internal`) +- `10.6.0.22`: Patroni 2 (`patroni2.internal`) +- `10.6.0.23`: Patroni 3 (`patroni3.internal`) ```plaintext global @@ -716,7 +714,7 @@ backend postgresql server patroni1.internal 10.6.0.21:5432 maxconn 100 check port 8008 server patroni2.internal 10.6.0.22:5432 maxconn 100 check port 8008 - server patroni3.internal 10.6.0.23.195:5432 maxconn 100 check port 8008 + server patroni3.internal 10.6.0.23:5432 maxconn 100 check port 8008 ``` Refer to your preferred Load Balancer's documentation for further guidance. diff --git a/doc/administration/geo/setup/external_database.md b/doc/administration/geo/setup/external_database.md index 13dbf9d1434..756e73f022f 100644 --- a/doc/administration/geo/setup/external_database.md +++ b/doc/administration/geo/setup/external_database.md @@ -8,7 +8,7 @@ type: howto # Geo with external PostgreSQL instances **(PREMIUM SELF)** This document is relevant if you are using a PostgreSQL instance that is *not -managed by Omnibus*. This includes cloud-managed instances like AWS RDS, or +managed by Omnibus*. This includes cloud-managed instances like Amazon RDS, or manually installed and configured PostgreSQL instances. NOTE: @@ -58,7 +58,7 @@ developed and tested. We aim to be compatible with most external To set up an external database, you can either: -- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example AWS RDS, bare metal not managed by Omnibus, and so on). +- Set up [streaming replication](https://www.postgresql.org/docs/12/warm-standby.html#STREAMING-REPLICATION-SLOTS) yourself (for example Amazon RDS, bare metal not managed by Omnibus, and so on). - Perform the Omnibus configuration manually as follows. #### Leverage your cloud provider's tools to replicate the primary database @@ -200,7 +200,7 @@ This is for the installation of extensions during installation and upgrades. As To setup an external tracking database, follow the instructions below: NOTE: -If you want to use AWS RDS as a tracking database, make sure it has access to +If you want to use Amazon RDS as a tracking database, make sure it has access to the secondary database. Unfortunately, just assigning the same security group is not enough as outbound rules do not apply to RDS PostgreSQL databases. Therefore, you need to explicitly add an inbound rule to the read-replica's security group allowing any TCP traffic from diff --git a/doc/administration/get_started.md b/doc/administration/get_started.md index 2455aecafea..bff2aee1d2d 100644 --- a/doc/administration/get_started.md +++ b/doc/administration/get_started.md @@ -44,7 +44,7 @@ Get started: **More resources** - Learn more about [running multiple Agile teams](https://www.youtube.com/watch?v=VR2r1TJCDew). -- Sync group memberships [by using LDAP](../administration/auth/ldap/index.md#group-sync). +- Sync group memberships [by using LDAP](../administration/auth/ldap/ldap_synchronization.md#group-sync). - Manage user access with inherited permissions. Use up to 20 levels of subgroups to organize both teams and projects. - Learn more about [inherited permissions](../user/project/members/index.md#inherited-membership). - View [nested category examples](../user/group/subgroups/index.md#overview). @@ -231,7 +231,7 @@ Rate limits also improve the security of your application. ### Configure rate limits for self-managed GitLab -You can make changes to your default rate limits from the Admin Area. For more information about configuration, see the [Admin Area page](../security/rate_limits.md#admin-area-settings). +You can make changes to your default rate limits from the Admin Area. For more information about configuration, see the [Admin Area page](../security/rate_limits.md#configurable-limits). - Define [issues rate limits](../user/admin_area/settings/rate_limit_on_issues_creation.md) to set a maximum number of issue creation requests per minute, per user. - Enforce [user and IP rate limits](../user/admin_area/settings/user_and_ip_rate_limits.md) for unauthenticated web requests. @@ -249,7 +249,7 @@ Rate limits also improve the security of your application. ### Configure rate limits for GitLab SaaS -You can make changes to your default rate limits from the Admin Area. For more information about configuration, see the [Admin Area page](../security/rate_limits.md#admin-area-settings). +You can make changes to your default rate limits from the Admin Area. For more information about configuration, see the [Admin Area page](../security/rate_limits.md#configurable-limits). - Review the rate limit page. - Read our [API page](../api/index.md) for more information about API and rate limiting. diff --git a/doc/administration/gitaly/praefect.md b/doc/administration/gitaly/praefect.md index d3ea71bc8d2..da456131a52 100644 --- a/doc/administration/gitaly/praefect.md +++ b/doc/administration/gitaly/praefect.md @@ -851,6 +851,10 @@ of choice already. Some examples include [HAProxy](https://www.haproxy.org/) Big-IP LTM, and Citrix Net Scaler. This documentation outlines what ports and protocols you need configure. +WARNING: +Long-running background jobs can maintain an idle connection with Praefect for up 6 hours. Set your load balancer timeout to be at least +6 hours long. + | LB Port | Backend Port | Protocol | |:--------|:-------------|:---------| | 2305 | 2305 | TCP | diff --git a/doc/administration/gitaly/troubleshooting.md b/doc/administration/gitaly/troubleshooting.md index 1f90ebb7565..d6d93b8af94 100644 --- a/doc/administration/gitaly/troubleshooting.md +++ b/doc/administration/gitaly/troubleshooting.md @@ -157,7 +157,7 @@ Confirm the following are all true: successfully creates the project but doesn't create the README. - When [tailing the logs](https://docs.gitlab.com/omnibus/settings/logs.html#tail-logs-in-a-console-on-the-server) on a Gitaly client and reproducing the error, you get `401` errors - when reaching the [`/api/v4/internal/allowed`](../../development/internal_api.md) endpoint: + when reaching the [`/api/v4/internal/allowed`](../../development/internal_api/index.md) endpoint: ```shell # api_json.log diff --git a/doc/administration/img/audit_events_v14_5.png b/doc/administration/img/audit_events_v14_5.png Binary files differnew file mode 100644 index 00000000000..57190463d05 --- /dev/null +++ b/doc/administration/img/audit_events_v14_5.png diff --git a/doc/administration/img/audit_log_v13_6.png b/doc/administration/img/audit_log_v13_6.png Binary files differdeleted file mode 100644 index 82ff3e9c87b..00000000000 --- a/doc/administration/img/audit_log_v13_6.png +++ /dev/null diff --git a/doc/administration/incoming_email.md b/doc/administration/incoming_email.md index 5b72583bc95..6b390cfc77a 100644 --- a/doc/administration/incoming_email.md +++ b/doc/administration/incoming_email.md @@ -155,6 +155,50 @@ Reply by email should now be working. 1. Find the `incoming_email` section in `config/gitlab.yml`, enable the feature and fill in the details for your specific IMAP server and email account (see [examples](#configuration-examples) below). +If you use systemd units to manage GitLab: + +1. Add `gitlab-mailroom.service` as a dependency to `gitlab.target`: + + ```shell + sudo systemctl edit gitlab.target + ``` + + In the editor that opens, add the following and save the file: + + ```plaintext + [Unit] + Wants=gitlab-mailroom.service + ``` + +1. If you run Redis and PostgreSQL on the same machine, you should add a + dependency on Redis. Run: + + ```shell + sudo systemctl edit gitlab-mailroom.service + ``` + + In the editor that opens, add the following and save the file: + + ```plaintext + [Unit] + Wants=redis-server.service + After=redis-server.service + ``` + +1. Start `gitlab-mailroom.service`: + + ```shell + sudo systemctl start gitlab-mailroom.service + ``` + +1. Verify that everything is configured correctly: + + ```shell + sudo -u git -H bundle exec rake gitlab:incoming_email:check RAILS_ENV=production + ``` + +If you use the SysV init script to manage GitLab: + 1. Enable `mail_room` in the init script at `/etc/default/gitlab`: ```shell diff --git a/doc/administration/index.md b/doc/administration/index.md index ee17edc35fc..53a3c305aab 100644 --- a/doc/administration/index.md +++ b/doc/administration/index.md @@ -114,7 +114,7 @@ Learn how to install, configure, update, and maintain your GitLab instance. - [Mattermost](../integration/mattermost/index.md): Integrate with [Mattermost](https://mattermost.com), an open source, private cloud workplace for web messaging. - [PlantUML](integration/plantuml.md): Create diagrams in AsciiDoc and Markdown documents created in snippets, wikis, and repositories. -- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from GitLab CI/CD [environments](../ci/environments/index.md#web-terminals). +- [Web terminals](integration/terminal.md): Provide terminal access to your applications deployed to Kubernetes from GitLab CI/CD [environments](../ci/environments/index.md#web-terminals-deprecated). ## User settings and permissions diff --git a/doc/administration/instance_limits.md b/doc/administration/instance_limits.md index a2729e60545..0b470146b14 100644 --- a/doc/administration/instance_limits.md +++ b/doc/administration/instance_limits.md @@ -15,9 +15,7 @@ performance, data, or could even exhaust the allocated resources for the applica Rate limits can be used to improve the security and durability of GitLab. -For example, one script can make thousands of web requests per second. Whether malicious, apathetic, or just a bug, your application and infrastructure may not be able to cope with the load. Rate limits can help to mitigate these types of attacks. - -Read more about [configuring rate limits](../security/rate_limits.md) in the Security documentation. +Read more about [configuring rate limits](../security/rate_limits.md). ### Issue creation @@ -128,16 +126,6 @@ This setting limits the import/export actions for groups and projects. Read more about [import/export rate limits](../user/admin_area/settings/import_export_rate_limits.md). -### Rack attack - -This method of rate limiting is cumbersome, but has some advantages. It allows -throttling of specific paths, and is also integrated into Git and container -registry requests. - -Read more about the [Rack Attack initializer](../security/rack_attack.md) method of setting rate limits. - -- **Default rate limit**: Disabled. - ### Member Invitations Limit the maximum daily member invitations allowed per group hierarchy. @@ -155,6 +143,9 @@ This only applies to project and group webhooks. Calls over the rate limit are logged into `auth.log`. +To set this limit for a self-managed installation, run the following in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + ```ruby # If limits don't exist for the default plan, you can create one with: # Plan.default.create_limits! @@ -222,10 +213,12 @@ When the number exceeds the limit the page displays an alert and links to a pagi > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51401) in GitLab 11.10. The number of pipelines that can be created in a single push is 4. -This is to prevent the accidental creation of pipelines when `git push --all` +This limit prevents the accidental creation of pipelines when `git push --all` or `git push --mirror` is used. -Read more in the [CI documentation](../ci/yaml/index.md#processing-git-pushes). +This limit does not affect any of the updated merge request pipelines. +All updated merge requests have a pipeline created when using +[pipelines for merge requests](../ci/pipelines/merge_request_pipelines.md). ## Retention of activity history @@ -397,6 +390,27 @@ Plan.default.actual_limits.update!(ci_project_subscriptions: 500) Set the limit to `0` to disable it. +### Limit the number of pipeline triggers + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33696) in GitLab 14.6. + +You can set a limit on the maximum number of pipeline triggers per project. This +limit is checked every time a new trigger is created. + +If a new trigger would cause the total number of pipeline triggers to exceed the +limit, the trigger is considered invalid. + +Set the limit to `0` to disable it. Defaults to `0` on self-managed instances. + +To set this limit to `100` on a self-managed installation, run the following in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +```ruby +Plan.default.actual_limits.update!(pipeline_triggers: 100) +``` + +This limit is [enabled on GitLab.com](../user/gitlab_com/index.md#gitlab-cicd). + ### Number of pipeline schedules > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29566) in GitLab 12.10. @@ -543,6 +557,7 @@ Plan.default.actual_limits.update!(pages_file_entries: 100) The total number of registered runners is limited at the group and project levels. Each time a new runner is registered, GitLab checks these limits against runners that have been active in the last 3 months. A runner's registration fails if it exceeds the limit for the scope determined by the runner registration token. +If the limit value is set to zero, the limit is disabled. - GitLab SaaS subscribers have different limits defined per plan, affecting all projects using that plan. - Self-managed GitLab Premium and Ultimate limits are defined by a default plan that affects all projects: @@ -590,25 +605,64 @@ Plan.default.actual_limits.update!(dast_profile_schedules: 50) ### Maximum size and depth of CI/CD configuration YAML files -The default maximum size of a CI/CD configuration YAML file is 1 megabyte and the default depth is 100. +The default maximum size of a CI/CD configuration YAML file is 1 megabyte and the +default depth is 100. + +You can change these limits in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): + +- To update the maximum YAML size, update `max_yaml_size_bytes` with the new value in megabytes: + + ```ruby + ApplicationSetting.update!(max_yaml_size_bytes: 2.megabytes) + ``` -You can change these limits in the [GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session). -Update `max_yaml_size_bytes` with the new value in megabytes: + The `max_yaml_size_bytes` value is not directly tied to the size of the YAML file, + but rather the memory allocated for the relevant objects. + +- To update the maximum YAML depth, update `max_yaml_depth` with the new value in megabytes: + + ```ruby + ApplicationSetting.update!(max_yaml_depth: 125) + ``` + +To disable this limitation entirely, disable the feature flag in the console: ```ruby -ApplicationSetting.update!(max_yaml_size_bytes: 2.megabytes) +Feature.disable(:ci_yaml_limit_size) ``` -Update `max_yaml_depth` with the new value in megabytes: +### Limit dotenv variables + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321552) in GitLab 14.5. + +You can set a limit on the maximum number of variables inside of a dotenv artifact. +This limit is checked every time a dotenv file is exported as an artifact. + +Set the limit to `0` to disable it. Defaults to `0` on self-managed instances. + +To set this limit to `100` on a self-managed instance, run the following command in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby -ApplicationSetting.update!(max_yaml_depth: 125) +Plan.default.actual_limits.update!(dotenv_variable_limit: 100) ``` -To disable this limitation entirely, disable the feature flag in the console: +This limit is [enabled on GitLab.com](../user/gitlab_com/index.md#gitlab-cicd). + +### Limit dotenv file size + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321552) in GitLab 14.5. + +You can set a limit on the maximum size of a dotenv artifact. This limit is checked +every time a dotenv file is exported as an artifact. + +Set the limit to `0` to disable it. Defaults to 5KB. + +To set this limit to 5KB on a self-managed installation, run the following in the +[GitLab Rails console](operations/rails_console.md#starting-a-rails-console-session): ```ruby -Feature.disable(:ci_yaml_limit_size) +Plan.default.actual_limits.update!(dotenv_size_limit: 5.kilobytes) ``` ## Instance monitoring and metrics @@ -837,6 +891,17 @@ Set the limit to `0` to allow any file size. When asking for versions of a given NuGet package name, the GitLab Package Registry returns a maximum of 300 versions. +## Dependency Proxy Limits + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6396) in GitLab 14.5. + +The maximum file size for an image cached in the +[Dependency Proxy](../user/packages/dependency_proxy/index.md) +varies by file type: + +- Image blob: 5 GB +- Image manifest: 10 MB + ## Branch retargeting on merge > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/320902) in GitLab 13.9. diff --git a/doc/administration/integration/terminal.md b/doc/administration/integration/terminal.md index 45b94781adc..07b9ba87d8e 100644 --- a/doc/administration/integration/terminal.md +++ b/doc/administration/integration/terminal.md @@ -4,12 +4,17 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Web terminals **(FREE)** +# Web terminals (DEPRECATED) **(FREE)** + +> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. With the introduction of the [Kubernetes integration](../../user/infrastructure/clusters/index.md), GitLab can store and use credentials for a Kubernetes cluster. GitLab uses these credentials to provide access to -[web terminals](../../ci/environments/index.md#web-terminals) for environments. +[web terminals](../../ci/environments/index.md#web-terminals-deprecated) for environments. NOTE: Only project maintainers and owners can access web terminals. diff --git a/doc/administration/load_balancer.md b/doc/administration/load_balancer.md index ca66791166e..60293e56202 100644 --- a/doc/administration/load_balancer.md +++ b/doc/administration/load_balancer.md @@ -7,7 +7,7 @@ type: reference # Load Balancer for multi-node GitLab **(FREE SELF)** -In an multi-node GitLab configuration, you need a load balancer to route +In a multi-node GitLab configuration, you need a load balancer to route traffic to the application servers. The specifics on which load balancer to use or the exact configuration is beyond the scope of GitLab documentation. We hope that if you're managing HA systems like GitLab you have a load balancer of @@ -69,7 +69,7 @@ for details on managing SSL certificates and configuring NGINX. | 443 | 443 | TCP or HTTPS (*1*) (*2*) | | 22 | 22 | TCP | -- (*1*): [Web terminal](../ci/environments/index.md#web-terminals) support requires +- (*1*): [Web terminal](../ci/environments/index.md#web-terminals-deprecated) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, this means your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the diff --git a/doc/administration/logs.md b/doc/administration/logs.md index a9fd698a525..bf74a96a627 100644 --- a/doc/administration/logs.md +++ b/doc/administration/logs.md @@ -752,7 +752,6 @@ Depending on your installation method, this file is located at: This log records: -- Information whenever [Rack Attack](../security/rack_attack.md) registers an abusive request. - Requests over the [Rate Limit](../user/admin_area/settings/rate_limits_on_raw_endpoints.md) on raw endpoints. - [Protected paths](../user/admin_area/settings/protected_paths.md) abusive requests. - In GitLab versions [12.3](https://gitlab.com/gitlab-org/gitlab/-/issues/29239) and later, diff --git a/doc/administration/maintenance_mode/index.md b/doc/administration/maintenance_mode/index.md index d5bcd132665..2d17062e955 100644 --- a/doc/administration/maintenance_mode/index.md +++ b/doc/administration/maintenance_mode/index.md @@ -119,7 +119,7 @@ For most JSON requests, POST, PUT, PATCH, and DELETE are blocked, and the API re | POST | `/admin/session`, `/admin/session/destroy` | To allow [Administrator mode for GitLab administrators](https://gitlab.com/groups/gitlab-org/-/epics/2158) | | POST | Paths ending with `/compare`| Git revision routes. | | POST | `.git/git-upload-pack` | To allow Git pull/clone. | -| POST | `/api/v4/internal` | [internal API routes](../../development/internal_api.md) | +| POST | `/api/v4/internal` | [internal API routes](../../development/internal_api/index.md) | | POST | `/admin/sidekiq` | To allow management of background jobs in the Admin UI | | POST | `/admin/geo` | To allow updating Geo Nodes in the administrator UI | | POST | `/api/v4/geo_replication`| To allow certain Geo-specific administrator UI actions on secondary sites | diff --git a/doc/administration/monitoring/performance/performance_bar.md b/doc/administration/monitoring/performance/performance_bar.md index 0befd9eac5b..d798feb71a9 100644 --- a/doc/administration/monitoring/performance/performance_bar.md +++ b/doc/administration/monitoring/performance/performance_bar.md @@ -114,7 +114,7 @@ for a given group: 1. On the left sidebar, select **Settings > Metrics and profiling** (`admin/application_settings/metrics_and_profiling`), and expand **Profiling - Performance bar**. -1. Click **Allow non-administrators to access to the performance bar**. +1. Click **Allow non-administrators access to the performance bar**. 1. In the **Allow access to members of the following group** field, provide the full path of the group allowed to access the performance. 1. Click **Save changes**. diff --git a/doc/administration/monitoring/prometheus/gitlab_metrics.md b/doc/administration/monitoring/prometheus/gitlab_metrics.md index 1d275010556..2f9c1e3bc9c 100644 --- a/doc/administration/monitoring/prometheus/gitlab_metrics.md +++ b/doc/administration/monitoring/prometheus/gitlab_metrics.md @@ -41,6 +41,7 @@ The following metrics are available: | `gitlab_cache_misses_total` | Counter | 10.2 | Cache read miss | `controller`, `action` | | `gitlab_cache_operation_duration_seconds` | Histogram | 10.2 | Cache access time | | | `gitlab_cache_operations_total` | Counter | 12.2 | Cache operations by controller or action | `controller`, `action`, `operation` | +| `gitlab_ci_pipeline_builder_scoped_variables_duration` | Histogram | 14.5 | Time in seconds it takes to create the scoped variables for a CI/CD job | `gitlab_ci_pipeline_creation_duration_seconds` | Histogram | 13.0 | Time in seconds it takes to create a CI/CD pipeline | | | `gitlab_ci_pipeline_size_builds` | Histogram | 13.1 | Total number of builds within a pipeline grouped by a pipeline source | `source` | | `job_waiter_started_total` | Counter | 12.9 | Number of batches of jobs started where a web request is waiting for the jobs to complete | `worker` | @@ -143,7 +144,7 @@ The following metrics are available: | `gitlab_snowplow_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emitted | | | `gitlab_snowplow_failed_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emission failures | | | `gitlab_snowplow_successful_events_total` | Counter | 14.1 | Total number of GitLab Snowplow product intelligence events emission successes | | -| `gitlab_ci_build_trace_errors_total` | Counter | 14.4 | Total amount of different error types on a build trace | `type` | +| `gitlab_ci_build_trace_errors_total` | Counter | 14.4 | Total amount of different error types on a build trace | `error_reason` | ## Metrics controlled by a feature flag @@ -189,9 +190,6 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_lfs_objects` | Gauge | 10.2 | Total number of LFS objects available on primary | `url` | | `geo_lfs_objects_synced` | Gauge | 10.2 | Number of LFS objects synced on secondary | `url` | | `geo_lfs_objects_failed` | Gauge | 10.2 | Number of LFS objects failed to sync on secondary | `url` | -| `geo_attachments` | Gauge | 10.2 | Total number of file attachments available on primary | `url` | -| `geo_attachments_synced` | Gauge | 10.2 | Number of attachments synced on secondary | `url` | -| `geo_attachments_failed` | Gauge | 10.2 | Number of attachments failed to sync on secondary | `url` | | `geo_last_event_id` | Gauge | 10.2 | Database ID of the latest event log entry on the primary | `url` | | `geo_last_event_timestamp` | Gauge | 10.2 | UNIX timestamp of the latest event log entry on the primary | `url` | | `geo_cursor_last_event_id` | Gauge | 10.2 | Last database ID of the event log processed by the secondary | `url` | @@ -200,7 +198,6 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_last_successful_status_check_timestamp` | Gauge | 10.2 | Last timestamp when the status was successfully updated | `url` | | `geo_lfs_objects_synced_missing_on_primary` | Gauge | 10.7 | Number of LFS objects marked as synced due to the file missing on the primary | `url` | | `geo_job_artifacts_synced_missing_on_primary` | Gauge | 10.7 | Number of job artifacts marked as synced due to the file missing on the primary | `url` | -| `geo_attachments_synced_missing_on_primary` | Gauge | 10.7 | Number of attachments marked as synced due to the file missing on the primary | `url` | | `geo_repositories_checksummed` | Gauge | 10.7 | Number of repositories checksummed on primary | `url` | | `geo_repositories_checksum_failed` | Gauge | 10.7 | Number of repositories failed to calculate the checksum on primary | `url` | | `geo_wikis_checksummed` | Gauge | 10.7 | Number of wikis checksummed on primary | `url` | @@ -275,6 +272,8 @@ configuration option in `gitlab.yml`. These metrics are served from the | `geo_uploads_synced` | Gauge | 14.1 | Number of uploads synced on secondary | `url` | | `geo_uploads_failed` | Gauge | 14.1 | Number of syncable uploads failed to sync on secondary | `url` | | `geo_uploads_registry` | Gauge | 14.1 | Number of uploads in the registry | `url` | +| `gitlab_sli:rails_request_apdex:total` | Counter | 14.4 | The number of request-apdex measurements, [more information the development documentation](../../../development/application_slis/rails_request_apdex.md) | `endpoint_id`, `feature_category`, `request_urgency` | +| `gitlab_sli:rails_request_apdex:success_total` | Counter | 14.4 | The number of succesful requests that met the target duration for their urgency. Devide by `gitlab_sli:rails_requests_apdex:total` to get a success ratio | `endpoint_id`, `feature_category`, `request_urgency` | ## Database load balancing metrics **(PREMIUM SELF)** diff --git a/doc/administration/monitoring/prometheus/index.md b/doc/administration/monitoring/prometheus/index.md index e04aad9c6b8..e86ca596955 100644 --- a/doc/administration/monitoring/prometheus/index.md +++ b/doc/administration/monitoring/prometheus/index.md @@ -345,6 +345,12 @@ memory, disk, and CPU utilization. [Read more about the node exporter](node_exporter.md). +### Puma exporter + +The Puma exporter allows you to measure various Puma metrics. + +[Read more about the Puma exporter](puma_exporter.md). + ### Redis exporter The Redis exporter allows you to measure various Redis metrics. diff --git a/doc/administration/monitoring/prometheus/puma_exporter.md b/doc/administration/monitoring/prometheus/puma_exporter.md index c348e74afaf..804c4243cfa 100644 --- a/doc/administration/monitoring/prometheus/puma_exporter.md +++ b/doc/administration/monitoring/prometheus/puma_exporter.md @@ -25,3 +25,5 @@ To enable the Puma exporter: for the changes to take effect. Prometheus begins collecting performance data from the Puma exporter exposed at `localhost:8083`. + +For more information on using Puma with GitLab, see [Puma](../../operations/puma.md). diff --git a/doc/administration/nfs.md b/doc/administration/nfs.md index f18c39af24a..2a2e9f05312 100644 --- a/doc/administration/nfs.md +++ b/doc/administration/nfs.md @@ -93,6 +93,27 @@ is moved to NFS. We are investigating the use of [fast lookup as the default](https://gitlab.com/groups/gitlab-org/-/epics/3104). +## Improving NFS performance with GitLab + +NFS performance with GitLab can in some cases be improved with +[direct Git access](gitaly/index.md#direct-access-to-git-in-gitlab) using +[Rugged](https://github.com/libgit2/rugged). + +From GitLab 12.1, GitLab automatically detects if Rugged can and should be used per storage. +If you previously enabled Rugged using the feature flag and you want to use automatic detection instead, +you must unset the feature flag: + +```shell +sudo gitlab-rake gitlab:features:unset_rugged +``` + +If the Rugged feature flag is explicitly set to either `true` or `false`, GitLab uses the value explicitly set. + +From GitLab 12.7, Rugged is only automatically enabled for use with Puma +if the [Puma thread count is set to `1`](../install/requirements.md#puma-settings). + +To use Rugged with a Puma thread count of more than `1`, enable Rugged using the [feature flag](../development/gitaly.md#legacy-rugged-code). + ## NFS server Installing the `nfs-kernel-server` package allows you to share directories with @@ -165,32 +186,6 @@ You may not need to disable NFS server delegation if you know you are using a ve the Linux kernel that has been fixed. That said, GitLab still encourages instance administrators to keep NFS server delegation disabled. -### Improving NFS performance with GitLab - -NFS performance with GitLab can in some cases be improved with -[direct Git access](gitaly/index.md#direct-access-to-git-in-gitlab) using -[Rugged](https://github.com/libgit2/rugged). - -NOTE: -From GitLab 12.1, it automatically detects if Rugged can and should be used per storage. - -If you previously enabled Rugged using the feature flag, you need to unset the feature flag by using: - -```shell -sudo gitlab-rake gitlab:features:unset_rugged -``` - -If the Rugged feature flag is explicitly set to either `true` or `false`, GitLab uses the value explicitly set. - -#### Improving NFS performance with Puma - -NOTE: -From GitLab 12.7, Rugged is not automatically enabled if Puma thread count is greater than `1`. - -If you want to use Rugged with Puma, [set Puma thread count to `1`](../install/requirements.md#puma-settings). - -If you want to use Rugged with Puma thread count more than `1`, Rugged can be enabled using the [feature flag](../development/gitaly.md#legacy-rugged-code). - ## NFS client The `nfs-common` provides NFS functionality without installing server components which @@ -232,7 +227,7 @@ Note there are several options that you should consider using: It's recommended that you use `hard` in your mount options, unless you have a specific reason to use `soft`. -On GitLab.com, we use `soft` because there were times when we had NFS servers +When GitLab.com used NFS, we used `soft` because there were times when we had NFS servers reboot and `soft` improved availability, but everyone's infrastructure is different. If your NFS is provided by on-premise storage arrays with redundant controllers, for example, you shouldn't need to worry about NFS server availability. diff --git a/doc/administration/object_storage.md b/doc/administration/object_storage.md index 6c77576cb27..8576b429213 100644 --- a/doc/administration/object_storage.md +++ b/doc/administration/object_storage.md @@ -2,7 +2,6 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Object storage **(FREE SELF)** @@ -77,7 +76,13 @@ Mattermost. See the [full table for a complete list](#storage-specific-configura However, backups can be configured with [server side encryption](../raketasks/backup_restore.md#s3-encrypted-buckets) separately. Enabling consolidated object storage enables object storage for all object -types. If you want to use local storage for specific object types, you can +types. If not all buckets are specified, `sudo gitlab-ctl reconfigure` may fail with the error like: + +```plaintext +Object storage for <object type> must have a bucket specified +``` + +If you want to use local storage for specific object types, you can [selectively disable object storages](#selectively-disabling-object-storage). Most types of objects, such as CI artifacts, LFS files, upload @@ -123,8 +128,8 @@ See the section on [ETag mismatch errors](#etag-mismatch) for more details. gitlab_rails['object_store']['objects']['pages']['bucket'] = '<pages>' ``` - For GitLab 9.4 or later, if you're using AWS IAM profiles, be sure to omit the - AWS access key and secret access key/value pairs. For example: + If you're using AWS IAM profiles, omit the AWS access key and secret access + key/value pairs. For example: ```ruby gitlab_rails['object_store']['connection'] = { @@ -558,7 +563,7 @@ supported by consolidated configuration form, refer to the following guides: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| **{dotted-circle}** No | | [Packages](packages/index.md#using-object-storage) (optional feature) | **{check-circle}** Yes | | [Dependency Proxy](packages/dependency_proxy.md#using-object-storage) (optional feature) **(PREMIUM SELF)** | **{check-circle}** Yes | -| [Pseudonymizer](pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | **{dotted-circle}** No | +| [Pseudonymizer](pseudonymizer.md) (optional feature) | **{dotted-circle}** No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | **{dotted-circle}** No | | [Terraform state files](terraform_state.md#using-object-storage) | **{check-circle}** Yes | | [GitLab Pages content](pages/index.md#using-object-storage) | **{check-circle}** Yes | diff --git a/doc/administration/operations/extra_sidekiq_processes.md b/doc/administration/operations/extra_sidekiq_processes.md index 31cbd6c457b..02cb7ad0bca 100644 --- a/doc/administration/operations/extra_sidekiq_processes.md +++ b/doc/administration/operations/extra_sidekiq_processes.md @@ -30,6 +30,11 @@ can be started. > - [Sidekiq cluster moved](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/181) to GitLab Free in 12.10. > - [Sidekiq cluster became default](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/4140) in GitLab 13.0. +When starting multiple processes, the number of processes should +equal (and **not** exceed) the number of CPU cores you want to +dedicate to Sidekiq. Each Sidekiq process can use only 1 CPU +core, subject to the available workload and concurrency settings. + To start multiple processes: 1. Using the `sidekiq['queue_groups']` array setting, specify how many processes to @@ -166,23 +171,41 @@ one only processes imports and the other processes all other queues: ## Number of threads -Each process defined under `sidekiq` starts with a +By default each process defined under `sidekiq` starts with a number of threads that equals the number of queues, plus one spare thread. For example, a process that handles the `process_commit` and `post_receive` queues uses three threads in total. -## Manage concurrency +These thread run inside a single Ruby process, and each process +can only use a single CPU core. The usefulness of threading depends +on the work having some external dependencies to wait on, like database queries or +HTTP requests. Most Sidekiq deployments benefit from this threading, and when +running fewer queues in a process, increasing the thread count might be +even more desirable to make the most effective use of CPU resources. + +### Manage thread counts explicitly + +The correct maximum thread count (also called concurrency) depends on the workload. +Typical values range from `1` for highly CPU-bound tasks to `15` or higher for mixed +low-priority work. A reasonable starting range is `15` to `25` for a non-specialized +deployment. + +You can find example values used by GitLab.com by searching for `concurrency:` in +[the Helm charts](https://gitlab.com/gitlab-com/gl-infra/k8s-workloads/gitlab-com/-/blob/master/releases/gitlab/values/gprd.yaml.gotmpl). +The values vary according to the work each specific deployment of Sidekiq does. +Any other specialized deployments with processes dedicated to specific queues should +have the concurrency tuned according to: +have the concurrency tuned according to: -When setting the maximum concurrency, keep in mind this normally should -not exceed the number of CPU cores available. The values in the examples -below are arbitrary and not particular recommendations. +- The CPU usage of each type of process. +- The throughput achieved. Each thread requires a Redis connection, so adding threads may increase Redis latency and potentially cause client timeouts. See the [Sidekiq documentation about Redis](https://github.com/mperham/sidekiq/wiki/Using-Redis) for more details. -### When running Sidekiq cluster (default) +#### When running Sidekiq cluster (default) Running Sidekiq cluster is the default in GitLab 13.0 and later. @@ -203,7 +226,7 @@ Running Sidekiq cluster is the default in GitLab 13.0 and later. the other. Setting `min_concurrency` to `0` disables the limit. For each queue group, let `N` be one more than the number of queues. The -concurrency factor are set to: +concurrency is set to: 1. `N`, if it's between `min_concurrency` and `max_concurrency`. 1. `max_concurrency`, if `N` exceeds this value. @@ -215,7 +238,7 @@ regardless of the number of queues. When `min_concurrency` is greater than `max_concurrency`, it is treated as being equal to `max_concurrency`. -### When running a single Sidekiq process +#### When running a single Sidekiq process Running a single Sidekiq process is the default in GitLab 12.10 and earlier. diff --git a/doc/administration/operations/fast_ssh_key_lookup.md b/doc/administration/operations/fast_ssh_key_lookup.md index 8aa5af4c2bf..c99f94589d7 100644 --- a/doc/administration/operations/fast_ssh_key_lookup.md +++ b/doc/administration/operations/fast_ssh_key_lookup.md @@ -26,11 +26,8 @@ indexed lookup in the GitLab database. This page describes how to enable the fas lookup of authorized SSH keys. WARNING: -OpenSSH version 6.9+ is required because -`AuthorizedKeysCommand` must be able to accept a fingerprint. These -instructions break installations that use older versions of OpenSSH, such as -those included with CentOS 6 as of September 2017. If you want to use this -feature for CentOS 6, follow [the instructions on how to build and install a custom OpenSSH package](#compiling-a-custom-version-of-openssh-for-centos-6) before continuing. +OpenSSH version 6.9+ is required because `AuthorizedKeysCommand` must be +able to accept a fingerprint. Check the version of OpenSSH on your server. ## Fast lookup is required for Geo **(PREMIUM)** @@ -132,100 +129,43 @@ This is a brief overview. Please refer to the above instructions for more contex 1. Remove the `AuthorizedKeysCommand` lines from `/etc/ssh/sshd_config` or from `/assets/sshd_config` if you are using Omnibus Docker. 1. Reload `sshd`: `sudo service sshd reload`. -## Compiling a custom version of OpenSSH for CentOS 6 +## Use `gitlab-sshd` instead of OpenSSH -Building a custom version of OpenSSH is not necessary for Ubuntu 16.04 users, -since Ubuntu 16.04 ships with OpenSSH 7.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299109) in GitLab 14.5. -It is also unnecessary for CentOS 7.4 users, as that version ships with -OpenSSH 7.4. If you are using CentOS 7.0 - 7.3, we strongly recommend that you -upgrade to CentOS 7.4 instead of following this procedure. This should be as -simple as running `yum update`. - -CentOS 6 users must build their own OpenSSH package to enable SSH lookups via -the database. The following instructions can be used to build OpenSSH 7.5: - -1. First, download the package and install the required packages: - - ```shell - sudo su - - cd /tmp - curl --remote-name "https://cdn.openbsd.org/pub/OpenBSD/OpenSSH/portable/openssh-7.5p1.tar.gz" - tar xzvf openssh-7.5p1.tar.gz - yum install rpm-build gcc make wget openssl-devel krb5-devel pam-devel libX11-devel xmkmf libXt-devel - ``` - -1. Prepare the build by copying files to the right place: - - ```shell - mkdir -p /root/rpmbuild/{SOURCES,SPECS} - cp ./openssh-7.5p1/contrib/redhat/openssh.spec /root/rpmbuild/SPECS/ - cp openssh-7.5p1.tar.gz /root/rpmbuild/SOURCES/ - cd /root/rpmbuild/SPECS - ``` - -1. Next, set the spec settings properly: - - ```shell - sed -i -e "s/%define no_gnome_askpass 0/%define no_gnome_askpass 1/g" openssh.spec - sed -i -e "s/%define no_x11_askpass 0/%define no_x11_askpass 1/g" openssh.spec - sed -i -e "s/BuildPreReq/BuildRequires/g" openssh.spec - ``` - -1. Build the RPMs: - - ```shell - rpmbuild -bb openssh.spec - ``` - -1. Ensure the RPMs were built: - - ```shell - ls -al /root/rpmbuild/RPMS/x86_64/ - ``` - - You should see something as the following: - - ```plaintext - total 1324 - drwxr-xr-x. 2 root root 4096 Jun 20 19:37 . - drwxr-xr-x. 3 root root 19 Jun 20 19:37 .. - -rw-r--r--. 1 root root 470828 Jun 20 19:37 openssh-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 490716 Jun 20 19:37 openssh-clients-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 17020 Jun 20 19:37 openssh-debuginfo-7.5p1-1.x86_64.rpm - -rw-r--r--. 1 root root 367516 Jun 20 19:37 openssh-server-7.5p1-1.x86_64.rpm - ``` - -1. Install the packages. OpenSSH packages replace `/etc/pam.d/sshd` - with their own versions, which may prevent users from logging in, so be sure - that the file is backed up and restored after installation: - - ```shell - timestamp=$(date +%s) - cp /etc/pam.d/sshd pam-ssh-conf-$timestamp - rpm -Uvh /root/rpmbuild/RPMS/x86_64/*.rpm - yes | cp pam-ssh-conf-$timestamp /etc/pam.d/sshd - ``` - -1. Verify the installed version. In another window, attempt to sign in to the - server: - - ```shell - ssh -v <your-centos-machine> +WARNING: +`gitlab-sshd` is in [**Alpha**](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha-beta-ga). +It is not ready for production use. + +`gitlab-sshd` is [a standalone SSH server](https://gitlab.com/gitlab-org/gitlab-shell/-/tree/main/internal/sshd) + written in Go. It is provided as a part of `gitlab-shell` package. It has a lower memory + use as a OpenSSH alternative and supports + [group access restriction by IP address](../../user/group/index.md) for applications + running behind the proxy. + +If you are considering switching from OpenSSH to `gitlab-sshd`, consider these concerns: + +- The `gitlab-sshd` component is only available for + [Cloud Native Helm Charts](https://docs.gitlab.com/charts/) deployments. +- `gitlab-sshd` supports the PROXY protocol. It can run behind proxy servers that rely + on it, such as HAProxy. +- `gitlab-sshd` does not share a SSH port with the system administrator's OpenSSH, + and requires a bind to port 22. +- `gitlab-sshd` **does not** support SSH certificates. + +To switch from OpenSSH to `gitlab-sshd`: + +1. Set the `gitlab-shell` charts `sshDaemon` option to + [`gitlab-sshd`](https://docs.gitlab.com/charts/charts/gitlab/gitlab-shell/index.html#installation-command-line-options). + For example: + + ```yaml + gitlab: + gitlab-shell: + sshDaemon: gitlab-sshd ``` - You should see a line that reads: "debug1: Remote protocol version 2.0, remote software version OpenSSH_7.5" - - If not, you may need to restart `sshd` (for example, `systemctl restart sshd.service`). - -1. *IMPORTANT!* Open a new SSH session to your server before exiting to make - sure everything is working! If you need to downgrade, simple install the - older package: - - ```shell - # Only run this if you run into a problem logging in - yum downgrade openssh-server openssh openssh-clients - ``` +1. Perform a Helm upgrade. ## SELinux support and limitations diff --git a/doc/administration/operations/moving_repositories.md b/doc/administration/operations/moving_repositories.md index 8aeaadc17e9..9cf7ac18c81 100644 --- a/doc/administration/operations/moving_repositories.md +++ b/doc/administration/operations/moving_repositories.md @@ -49,8 +49,8 @@ until the move has completed. To move repositories: -1. Ensure all storages are accessible to the GitLab instance. In this example, these are - `<original_storage_name>` and `<cluster_storage_name>`. +1. Ensure all [local and cluster storages](../gitaly/configure_gitaly.md#mixed-configuration) are accessible to the GitLab instance. In + this example, these are `<original_storage_name>` and `<cluster_storage_name>`. 1. [Configure repository storage weights](../repository_storage_paths.md#configure-where-new-repositories-are-stored) so that the new storages receives all new projects. This stops new projects from being created on existing storages while the migration is in progress. diff --git a/doc/administration/operations/puma.md b/doc/administration/operations/puma.md index 775761d655d..f1f02b606f5 100644 --- a/doc/administration/operations/puma.md +++ b/doc/administration/operations/puma.md @@ -224,3 +224,8 @@ in Puma when using the Linux package, and which ones have no corresponding count | `unicorn['exporter_enabled']` | `puma['exporter_enabled']` | | `unicorn['exporter_address']` | `puma['exporter_address']` | | `unicorn['exporter_port']` | `puma['exporter_port']` | + +## Puma exporter + +You can use the Puma exporter to measure various Puma metrics. For more information, see +[Puma exporter](../monitoring/prometheus/puma_exporter.md). diff --git a/doc/administration/operations/ssh_certificates.md b/doc/administration/operations/ssh_certificates.md index 77dc4eb180b..4fcc5d7c916 100644 --- a/doc/administration/operations/ssh_certificates.md +++ b/doc/administration/operations/ssh_certificates.md @@ -19,10 +19,8 @@ In such setups some external automated process is needed to constantly upload the new keys to GitLab. WARNING: -OpenSSH version 6.9+ is required because that version -introduced the `AuthorizedPrincipalsCommand` configuration option. If -using CentOS 6, you can [follow these instructions](fast_ssh_key_lookup.md#compiling-a-custom-version-of-openssh-for-centos-6) -to compile an up-to-date version. +OpenSSH version 6.9+ is required because `AuthorizedKeysCommand` must be +able to accept a fingerprint. Check the version of OpenSSH on your server. ## Why use OpenSSH certificates? diff --git a/doc/administration/package_information/deprecated_os.md b/doc/administration/package_information/deprecated_os.md index 35b333241b2..7234d68e4b2 100644 --- a/doc/administration/package_information/deprecated_os.md +++ b/doc/administration/package_information/deprecated_os.md @@ -64,6 +64,7 @@ The following lists the currently supported OSs and their possible EOL dates. | Debian 9 | GitLab CE / GitLab EE 9.3.0 | amd64 | 2022 | <https://wiki.debian.org/DebianReleases#Production_Releases> | | Debian 10 | GitLab CE / GitLab EE 12.2.0 | amd64, arm64 | TBD | <https://wiki.debian.org/DebianReleases#Production_Releases> | | OpenSUSE 15.2 | GitLab CE / GitLab EE 13.11.0 | x86_64, aarch64 | Dec 2021 | <https://en.opensuse.org/Lifetime> | +| OpenSUSE 15.3 | GitLab CE / GitLab EE 14.5.0 | x86_64, aarch64 | Nov 2022 | <https://en.opensuse.org/Lifetime> | | SLES 12 | GitLab EE 9.0.0 | x86_64 | Oct 2027 | <https://www.suse.com/lifecycle/> | | Ubuntu 18.04 | GitLab CE / GitLab EE 10.7.0 | amd64 | April 2023 | <https://wiki.ubuntu.com/Releases> | | Ubuntu 20.04 | GitLab CE / GitLab EE 13.2.0 | amd64, arm64 | April 2025 | <https://wiki.ubuntu.com/Releases> | diff --git a/doc/administration/package_information/deprecation_policy.md b/doc/administration/package_information/deprecation_policy.md index 80ce72d54f5..d45c2ea3127 100644 --- a/doc/administration/package_information/deprecation_policy.md +++ b/doc/administration/package_information/deprecation_policy.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Deprecation policy **(FREE SELF)** @@ -35,9 +35,11 @@ This section lists steps necessary for deprecating and removing configuration. We can differentiate two different types of configuration: -- Sensitive: Configuration that can cause major service outage ( Data integrity, - installation integrity, preventing users from reaching the installation, etc.) -- Regular: Configuration that can make a feature unavailable but still makes the installation useable ( Change in default project/group settings, miscommunication with other components and similar ) +- Sensitive: Configuration that can cause major service outage (like data integrity, + installation integrity, or preventing users from reaching the installation) +- Regular: Configuration that can make a feature unavailable but still makes the + installation useable (like a change in default project/group settings, or + miscommunication with other components) We also need to differentiate deprecation and removal procedure. diff --git a/doc/administration/pages/index.md b/doc/administration/pages/index.md index 8a0d3f552bf..163eb5388b6 100644 --- a/doc/administration/pages/index.md +++ b/doc/administration/pages/index.md @@ -237,6 +237,7 @@ control over how the Pages daemon runs and serves content in your environment. | `log_verbose` | Verbose logging, true/false. | | `propagate_correlation_id` | Set to true (false by default) to re-use existing Correlation ID from the incoming request header `X-Request-ID` if present. If a reverse proxy sets this header, the value is propagated in the request chain. | | `max_connections` | Limit on the number of concurrent connections to the HTTP, HTTPS or proxy listeners. | +| `max_uri_length` | The maximum length of URIs accepted by GitLab Pages. Set to 0 for unlimited length. [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/659) in GitLab 14.5. | `metrics_address` | The address to listen on for metrics requests. | | `redirect_http` | Redirect pages from HTTP to HTTPS, true/false. | | `sentry_dsn` | The address for sending Sentry crash reporting to. | @@ -258,8 +259,8 @@ control over how the Pages daemon runs and serves content in your environment. | `FF_ENABLE_REDIRECTS` | Feature flag to enable/disable redirects (enabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#feature-flag-for-redirects) for more information. | | `FF_ENABLE_PLACEHOLDERS` | Feature flag to enable/disable rewrites (disabled by default). Read the [redirects documentation](../../user/project/pages/redirects.md#feature-flag-for-rewrites) for more information. | | `use_legacy_storage` | Temporarily-introduced parameter allowing to use legacy domain configuration source and storage. [Removed in 14.3](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6166). | - ---- +| `rate_limit_source_ip` | Rate limit per source IP in number of requests per second. Set to `0` to disable this feature. | +| `rate_limit_source_ip_burst` | Rate limit per source IP maximum burst allowed per second. | ## Advanced configuration @@ -647,7 +648,7 @@ To override the global maximum pages size for a specific group: ## Running GitLab Pages on a separate server You can run the GitLab Pages daemon on a separate server to decrease the load on -your main application server. +your main application server. This configuration does not support mutual TLS (mTLS). See the [corresponding feature proposal](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/548) for more information. To configure GitLab Pages on a separate server: @@ -1031,6 +1032,38 @@ GitLab Pages are part of the [regular backup](../../raketasks/backup_restore.md) You should strongly consider running GitLab Pages under a different hostname than GitLab to prevent XSS attacks. +### Rate limits + +> [Introduced](https://gitlab.com/gitlab-org/gitlab-pages/-/issues/631) in GitLab 14.5. + +You can enforce source-IP rate limits to help minimize the risk of a Denial of Service (DoS) attack. GitLab Pages +uses a [token bucket algorithm](https://en.wikipedia.org/wiki/Token_bucket) to enforce rate limiting. By default, +requests that exceed the specified limits are reported but not rejected. + +Source-IP rate limits are enforced using the following: + +- `rate_limit_source_ip`: Set the maximum threshold in number of requests per second. Set to 0 to disable this feature. +- `rate_limit_source_ip_burst`: Sets the maximum threshold of number of requests allowed in an initial outburst of requests. + For example, when you load a web page that loads a number of resources at the same time. + +#### Enable source-IP rate limits + +1. Set rate limits in `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['rate_limit_source_ip'] = 20.0 + gitlab_pages['rate_limit_source_ip_burst'] = 600 + ``` + +1. To reject requests that exceed the specified limits, enable the `FF_ENABLE_RATE_LIMITER` feature flag in + `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab_pages['env'] = {'FF_ENABLE_RATE_LIMITER' => 'true'} + ``` + +1. [Reconfigure GitLab](../restart_gitlab.md#omnibus-gitlab-reconfigure). + <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/administration/pages/source.md b/doc/administration/pages/source.md index 278d792052a..3a277204d21 100644 --- a/doc/administration/pages/source.md +++ b/doc/administration/pages/source.md @@ -132,11 +132,11 @@ The Pages daemon doesn't listen to the outside world. https: false artifacts_server: false external_http: ["127.0.0.1:8090"] - secret_file: /home/git/gitlab/gitlab-pages/gitlab-pages-secret + secret_file: /home/git/gitlab/gitlab-pages-secret ``` 1. Add the following configuration file to - `/home/git/gitlab/gitlab-pages/gitlab-pages.conf`, and be sure to change + `/home/git/gitlab-pages/gitlab-pages.conf`, and be sure to change `example.io` to the FQDN from which you want to serve GitLab Pages and `gitlab.example.com` to the URL of your GitLab instance: @@ -159,12 +159,27 @@ The Pages daemon doesn't listen to the outside world. sudo -u git -H openssl rand -base64 32 > /home/git/gitlab/gitlab-pages-secret ``` -1. Edit `/etc/default/gitlab` and set `gitlab_pages_enabled` to `true` in - order to enable the pages daemon: +1. To enable the pages daemon: - ```ini - gitlab_pages_enabled=true - ``` + - If your system uses systemd as init, run: + + ```shell + sudo systemctl edit gitlab.target + ``` + + In the editor that opens, add the following and save the file: + + ```plaintext + [Unit] + Wants=gitlab-pages.service + ``` + + - If your system uses SysV init instead, edit `/etc/default/gitlab` and set + `gitlab_pages_enabled` to `true`: + + ```ini + gitlab_pages_enabled=true + ``` 1. Copy the `gitlab-pages` NGINX configuration file: diff --git a/doc/administration/postgresql/external.md b/doc/administration/postgresql/external.md index 8f0fe0ace87..67c5448a8a0 100644 --- a/doc/administration/postgresql/external.md +++ b/doc/administration/postgresql/external.md @@ -4,7 +4,7 @@ group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Configure GitLab using an external PostgreSQL service +# Configure GitLab using an external PostgreSQL service **(FREE SELF)** If you're hosting GitLab on a cloud provider, you can optionally use a managed service for PostgreSQL. For example, AWS offers a managed Relational diff --git a/doc/administration/postgresql/index.md b/doc/administration/postgresql/index.md index bce78bbccff..15425a6d9f2 100644 --- a/doc/administration/postgresql/index.md +++ b/doc/administration/postgresql/index.md @@ -2,26 +2,15 @@ stage: Enablement group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- -# Configuring PostgreSQL for scaling +# Configuring PostgreSQL for scaling **(FREE SELF)** In this section, you'll be guided through configuring a PostgreSQL database to be used with GitLab in one of our [reference architectures](../reference_architectures/index.md). There are essentially three setups to choose from. -## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM SELF)** - -This setup is for when you have installed GitLab using the -[Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee). - -All the tools that are needed like PostgreSQL, PgBouncer, and Patroni are bundled in -the package, so you can it to set up the whole PostgreSQL infrastructure (primary, replica). - -[> Read how to set up PostgreSQL replication and failover using Omnibus GitLab](replication_and_failover.md) - -## Standalone PostgreSQL using Omnibus GitLab **(FREE SELF)** +## Standalone PostgreSQL using Omnibus GitLab This setup is for when you have installed the [Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE), @@ -29,7 +18,7 @@ to use the bundled PostgreSQL having only its service enabled. [> Read how to set up a standalone PostgreSQL instance using Omnibus GitLab](standalone.md) -## Provide your own PostgreSQL instance **(FREE SELF)** +## Provide your own PostgreSQL instance This setup is for when you have installed GitLab using the [Omnibus GitLab packages](https://about.gitlab.com/install/) (CE or EE), @@ -37,3 +26,13 @@ or installed it [from source](../../install/installation.md), but you want to us your own external PostgreSQL server. [> Read how to set up an external PostgreSQL instance](external.md) + +## PostgreSQL replication and failover with Omnibus GitLab **(PREMIUM SELF)** + +This setup is for when you have installed GitLab using the +[Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee). + +All the tools that are needed like PostgreSQL, PgBouncer, and Patroni are bundled in +the package, so you can use it to set up the whole PostgreSQL infrastructure (primary, replica). + +[> Read how to set up PostgreSQL replication and failover using Omnibus GitLab](replication_and_failover.md) diff --git a/doc/administration/postgresql/pgbouncer.md b/doc/administration/postgresql/pgbouncer.md index e215622bbc7..e5fef61540a 100644 --- a/doc/administration/postgresql/pgbouncer.md +++ b/doc/administration/postgresql/pgbouncer.md @@ -84,7 +84,7 @@ This content has been moved to a [new location](replication_and_failover.md#conf Do not backup or restore GitLab through a PgBouncer connection: it causes a GitLab outage. -[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer). +[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer). ## Enable Monitoring @@ -172,7 +172,7 @@ ote_pid | tls Some database changes have to be done directly, and not through PgBouncer. -Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer) +Read more about the affected tasks: [database restores](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer) and [GitLab upgrades](../../update/zero_downtime.md#use-postgresql-ha). 1. To find the primary node, run the following on a database node: diff --git a/doc/administration/postgresql/replication_and_failover.md b/doc/administration/postgresql/replication_and_failover.md index dc569a81abf..01fe4bf64ba 100644 --- a/doc/administration/postgresql/replication_and_failover.md +++ b/doc/administration/postgresql/replication_and_failover.md @@ -501,7 +501,7 @@ in the Troubleshooting section before proceeding. Do not backup or restore GitLab through a PgBouncer connection: this causes a GitLab outage. -[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#backup-and-restore-for-installations-using-pgbouncer). +[Read more about this and how to reconfigure backups](../../raketasks/backup_restore.md#back-up-and-restore-for-installations-using-pgbouncer). ### Ensure GitLab is running diff --git a/doc/administration/pseudonymizer.md b/doc/administration/pseudonymizer.md index da3a2e4b34c..bd6982bea12 100644 --- a/doc/administration/pseudonymizer.md +++ b/doc/administration/pseudonymizer.md @@ -6,33 +6,38 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Pseudonymizer **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/5532) in GitLab 11.1. +Your GitLab database contains sensitive information. To protect sensitive information +when you run analytics on your database, you can use the Pseudonymizer service, which: -As the GitLab database hosts sensitive information, using it unfiltered for analytics -implies high security requirements. To help alleviate this constraint, the Pseudonymizer -service is used to export GitLab data in a pseudonymized way. +1. Uses `HMAC(SHA256)` to mutate fields containing sensitive information. +1. Preserves references (referential integrity) between fields. +1. Exports your GitLab data, scrubbed of sensitive material. WARNING: -This process is not impervious. If the source data is available, it's possible for -a user to correlate data to the pseudonymized version. +If the source data is available, users can compare and correlate the scrubbed data +with the original. -The Pseudonymizer currently uses `HMAC(SHA256)` to mutate fields that shouldn't -be textually exported. This ensures that: +To generate a pseudonymized data set: -- the end-user of the data source cannot infer/revert the pseudonymized fields -- the referential integrity is maintained +1. [Configure Pseudonymizer](#configure-pseudonymizer) fields and output location. +1. [Enable Pseudonymizer data collection](#enable-pseudonymizer-data-collection). +1. Optional. [Generate a data set manually](#generate-data-set-manually). -## Configuration +## Configure Pseudonymizer -To configure the Pseudonymizer, you need to: +To use the Pseudonymizer, configure both the fields you want to anonymize, and the location to +store the scrubbed data: -- Provide a manifest file that describes which fields should be included or - pseudonymized ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab/-/tree/master/config/pseudonymizer.yml)). - A default manifest is provided with the GitLab installation, using a relative file path that resolves from the Rails root. - Alternatively, you can use an absolute file path. -- Use an object storage and specify the connection parameters in the `pseudonymizer.upload.connection` configuration option. - -[Read more about using object storage with GitLab](object_storage.md). +1. **Create a manifest file**: This file describes the fields to include or pseudonymize. + - **Default manifest** - GitLab provides a default manifest in your GitLab installation + ([example `manifest.yml` file](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/pseudonymizer.yml)). + To use the example manifest file, use the `config/pseudonymizer.yml` relative path + when you configure connection parameters. + - **Custom manifest** - To use a custom manifest file, use the absolute path to + the file when you configure the connection parameters. +1. **Configure connection parameters**: In the configuration method appropriate for + your version of GitLab, specify the [object storage](object_storage.md) + connection parameters (`pseudonymizer.upload.connection`). **For Omnibus installations:** @@ -50,7 +55,7 @@ To configure the Pseudonymizer, you need to: } ``` - If you are using AWS IAM profiles, be sure to omit the AWS access key and secret access key/value pairs. + If you are using AWS IAM profiles, omit the AWS access key and secret access key/value pairs. ```ruby gitlab_rails['pseudonymizer_upload_connection'] = { @@ -85,24 +90,34 @@ To configure the Pseudonymizer, you need to: 1. Save the file and [restart GitLab](restart_gitlab.md#installations-from-source) for the changes to take effect. -## Usage +## Enable Pseudonymizer data collection + +To enable data collection: + +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Metrics and Profiling**, then expand + **Pseudonymizer data collection**. +1. Select **Enable Pseudonymizer data collection**. +1. Select **Save changes**. -You can optionally run the Pseudonymizer using the following environment variables: +## Generate data set manually -- `PSEUDONYMIZER_OUTPUT_DIR` - where to store the output CSV files (defaults to `/tmp`) -- `PSEUDONYMIZER_BATCH` - the batch size when querying the DB (defaults to `100000`) +You can also run the Pseudonymizer manually: -```shell -## Omnibus -sudo gitlab-rake gitlab:db:pseudonymizer +1. Set these environment variables: + - `PSEUDONYMIZER_OUTPUT_DIR` - Where to store the output CSV files. Defaults to `/tmp`. + These commands produce CSV files that can be quite large. Make sure the directory + can store a file at least 10% of the size of your database. + - `PSEUDONYMIZER_BATCH` - The batch size when querying the database. Defaults to `100000`. +1. Run the command appropriate for your application: + - **Omnibus GitLab**: + `sudo gitlab-rake gitlab:db:pseudonymizer` + - **Installations from source**: + `sudo -u git -H bundle exec rake gitlab:db:pseudonymizer RAILS_ENV=production` -## Source -sudo -u git -H bundle exec rake gitlab:db:pseudonymizer RAILS_ENV=production -``` +After you run the command, upload the output CSV files to your configured object +storage. After the upload completes, delete the output file from the local disk. -This produces some CSV files that might be very large, so make sure the -`PSEUDONYMIZER_OUTPUT_DIR` has sufficient space. As a rule of thumb, at least -10% of the database size is recommended. +## Related topics -After the pseudonymizer has run, the output CSV files should be uploaded to the -configured object storage and deleted from the local disk. +- [Using object storage with GitLab](object_storage.md). diff --git a/doc/administration/raketasks/github_import.md b/doc/administration/raketasks/github_import.md index f29e2a6c7f6..0cdfd1c28ff 100644 --- a/doc/administration/raketasks/github_import.md +++ b/doc/administration/raketasks/github_import.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitHub import **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/10308) in GitLab 9.1. - To retrieve and import GitHub repositories, you need a [GitHub personal access token](https://github.com/settings/tokens). A username should be passed as the second argument to the Rake task, which becomes the owner of the project. You can resume an import diff --git a/doc/administration/raketasks/ldap.md b/doc/administration/raketasks/ldap.md index 585d254e41d..62ddd9be2b3 100644 --- a/doc/administration/raketasks/ldap.md +++ b/doc/administration/raketasks/ldap.md @@ -38,13 +38,13 @@ rake gitlab:ldap:check[50] > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/14735) in GitLab 12.2. -The following task runs a [group sync](../auth/ldap/index.md#group-sync) immediately. This is valuable -when you'd like to update all configured group memberships against LDAP without +The following task runs a [group sync](../auth/ldap/ldap_synchronization.md#group-sync) immediately. +This is valuable when you'd like to update all configured group memberships against LDAP without waiting for the next scheduled group sync to be run. NOTE: If you'd like to change the frequency at which a group sync is performed, -[adjust the cron schedule](../auth/ldap/index.md#adjust-ldap-group-sync-schedule) +[adjust the cron schedule](../auth/ldap/ldap_synchronization.md#adjust-ldap-group-sync-schedule) instead. **Omnibus Installation** diff --git a/doc/administration/raketasks/project_import_export.md b/doc/administration/raketasks/project_import_export.md index e0ca7bfdeaf..13f8dfdccc2 100644 --- a/doc/administration/raketasks/project_import_export.md +++ b/doc/administration/raketasks/project_import_export.md @@ -6,9 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Project import/export administration **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/3050) in GitLab 8.9. -> - From GitLab 11.3, import/export can use object storage automatically. - GitLab provides Rake tasks relating to project import and export. For more information, see: - [Project import/export documentation](../../user/project/settings/import_export.md). diff --git a/doc/administration/reference_architectures/10k_users.md b/doc/administration/reference_architectures/10k_users.md index e731085b0ce..9c3c33e1fa8 100644 --- a/doc/administration/reference_architectures/10k_users.md +++ b/doc/administration/reference_architectures/10k_users.md @@ -12,9 +12,10 @@ full list of reference architectures, see > - **Supported users (approximate):** 10,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) -> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS -> - **[Latest 10k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)** +> - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) +> - **Performance tested daily with the [GitLab Performance Tool](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Test requests per second (RPS) rates:** API: 200 RPS, Web: 20 RPS, Git (Pull): 20 RPS, Git (Push): 4 RPS +> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k)** | Service | Nodes | Configuration | GCP | AWS | Azure | |-----------------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| @@ -36,7 +37,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -279,7 +280,7 @@ The basic ports to be used are shown in the table below. | 443 | 443 | TCP or HTTPS (*1*) (*2*) | | 22 | 22 | TCP | -- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals-deprecated) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, this means your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the @@ -2085,7 +2086,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | +| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -2199,7 +2200,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/1k_users.md b/doc/administration/reference_architectures/1k_users.md index ae832c2226f..5488d8d33a6 100644 --- a/doc/administration/reference_architectures/1k_users.md +++ b/doc/administration/reference_architectures/1k_users.md @@ -20,8 +20,9 @@ many organizations. > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). > - **Cloud Native Hybrid:** No. For a cloud native hybrid environment, you > can follow a [modified hybrid reference architecture](#cloud-native-hybrid-reference-architecture-with-helm-charts). -> - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS -> - **[Latest 1k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)** +> - **Performance tested daily with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Test requests per second (RPS) rates:** API: 20 RPS, Web: 2 RPS, Git (Pull): 2 RPS, Git (Push): 1 RPS +> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/1k)** | Users | Configuration | GCP | AWS | Azure | |--------------|-------------------------|----------------|--------------|----------| diff --git a/doc/administration/reference_architectures/25k_users.md b/doc/administration/reference_architectures/25k_users.md index 267f81efd91..25cafbe667b 100644 --- a/doc/administration/reference_architectures/25k_users.md +++ b/doc/administration/reference_architectures/25k_users.md @@ -12,9 +12,10 @@ full list of reference architectures, see > - **Supported users (approximate):** 25,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) -> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS -> - **[Latest 25k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)** +> - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) +> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Test requests per second (RPS) rates:** API: 500 RPS, Web: 50 RPS, Git (Pull): 50 RPS, Git (Push): 10 RPS +> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/25k)** | Service | Nodes | Configuration | GCP | AWS | Azure | |---------------------------------------------------|-------------|-------------------------|------------------|--------------|-----------| @@ -36,7 +37,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -282,7 +283,7 @@ The basic ports to be used are shown in the table below. | 443 | 443 | TCP or HTTPS (*1*) (*2*) | | 22 | 22 | TCP | -- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals-deprecated) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, this means your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the @@ -2091,7 +2092,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | +| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -2199,7 +2200,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/2k_users.md b/doc/administration/reference_architectures/2k_users.md index 23b15b563f7..e619294704f 100644 --- a/doc/administration/reference_architectures/2k_users.md +++ b/doc/administration/reference_architectures/2k_users.md @@ -14,8 +14,9 @@ For a full list of reference architectures, see > - **High Availability:** No. For a highly-available environment, you can > follow a modified [3K reference architecture](3k_users.md#supported-modifications-for-lower-user-counts-ha). > - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS -> - **[Latest 2k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)** +> - **Performance tested daily with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Test requests per second (RPS) rates:** API: 40 RPS, Web: 4 RPS, Git (Pull): 4 RPS, Git (Push): 1 RPS +> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k)** | Service | Nodes | Configuration | GCP | AWS | Azure | |------------------------------------------|--------|-------------------------|-----------------|--------------|----------| @@ -29,7 +30,7 @@ For a full list of reference architectures, see | NFS server (optional, not recommended) | 1 | 4 vCPU, 3.6 GB memory | `n1-highcpu-4` | `c5.xlarge` | `F4s v2` | <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run as reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run as reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -176,7 +177,7 @@ table: | 443 | 443 | TCP or HTTPS (*1*) (*2*) | | 22 | 22 | TCP | -- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals-deprecated) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. For @@ -352,7 +353,7 @@ Omnibus: ```ruby ## Enable Redis redis['enable'] = true - + # Avoid running unnecessary services on the Redis server gitaly['enable'] = false postgresql['enable'] = false @@ -922,7 +923,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | +| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -1027,7 +1028,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. <!-- markdownlint-enable MD029 --> diff --git a/doc/administration/reference_architectures/3k_users.md b/doc/administration/reference_architectures/3k_users.md index 575dd22b729..9332ae8d271 100644 --- a/doc/administration/reference_architectures/3k_users.md +++ b/doc/administration/reference_architectures/3k_users.md @@ -22,9 +22,10 @@ For a full list of reference architectures, see > - **Supported users (approximate):** 3,000 > - **High Availability:** Yes, although [Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution -> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS -> - **[Latest 3k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)** +> - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) +> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Test requests per second (RPS) rates:** API: 60 RPS, Web: 6 RPS, Git (Pull): 6 RPS, Git (Push): 1 RPS +> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k)** | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-----------------------|-----------------|--------------|----------| @@ -45,7 +46,7 @@ For a full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -296,7 +297,7 @@ The basic ports to be used are shown in the table below. | 443 | 443 | TCP or HTTPS (*1*) (*2*) | | 22 | 22 | TCP | -- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals-deprecated) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, this means your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the @@ -2039,7 +2040,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | +| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -2106,7 +2107,7 @@ but with smaller performance requirements, several modifications can be consider - GitLab Rails and Sidekiq: Stateless services don't have a minimum node count. Two are enough for redundancy. - Gitaly and Praefect: A quorum is not strictly necessary. Two Gitaly nodes and two Praefect nodes are enough for redundancy. - Running select components in reputable Cloud PaaS solutions: Select components of the GitLab setup can instead be run on Cloud Provider PaaS solutions. By doing this, additional dependent components can also be removed: - - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or AWS RDS. In this setup, the PgBouncer and Consul nodes are no longer required: + - PostgreSQL: Can be run on reputable Cloud PaaS solutions such as Google Cloud SQL or Amazon RDS. In this setup, the PgBouncer and Consul nodes are no longer required: - Consul may still be desired if [Prometheus](../monitoring/prometheus/index.md) auto discovery is a requirement, otherwise you would need to [manually add scrape configurations](../monitoring/prometheus/index.md#adding-custom-scrape-configurations) for all nodes. - As Redis Sentinel runs on the same box as Consul in this architecture, it may need to be run on a separate box if Redis is still being run via Omnibus. - Redis: Can be run on reputable Cloud PaaS solutions such as Google Memorystore and AWS ElastiCache. In this setup, the Redis Sentinel is no longer required. @@ -2170,7 +2171,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/50k_users.md b/doc/administration/reference_architectures/50k_users.md index be44f464e7e..bbdf798d9ad 100644 --- a/doc/administration/reference_architectures/50k_users.md +++ b/doc/administration/reference_architectures/50k_users.md @@ -12,9 +12,10 @@ full list of reference architectures, see > - **Supported users (approximate):** 50,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) -> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS -> - **[Latest 50k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)** +> - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) +> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Test requests per second (RPS) rates:** API: 1000 RPS, Web: 100 RPS, Git (Pull): 100 RPS, Git (Push): 20 RPS +> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k)** | Service | Nodes | Configuration | GCP | AWS | Azure | |---------------------------------------------------|-------------|-------------------------|------------------|---------------|-----------| @@ -36,7 +37,7 @@ full list of reference architectures, see <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. @@ -288,7 +289,7 @@ The basic ports to be used are shown in the table below. | 443 | 443 | TCP or HTTPS (*1*) (*2*) | | 22 | 22 | TCP | -- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals-deprecated) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, this means your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the @@ -2105,7 +2106,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | +| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | @@ -2213,7 +2214,7 @@ services where applicable): <!-- Disable ordered list rule https://github.com/DavidAnson/markdownlint/blob/main/doc/Rules.md#md029---ordered-list-item-prefix --> <!-- markdownlint-disable MD029 --> -1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and AWS RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. +1. Can be optionally run on reputable third-party external PaaS PostgreSQL solutions. Google Cloud SQL and Amazon RDS are known to work, however Azure Database for PostgreSQL is [not recommended](https://gitlab.com/gitlab-org/quality/reference-architectures/-/issues/61) due to performance issues. Consul is primarily used for PostgreSQL high availability so can be ignored when using a PostgreSQL PaaS setup. However it is also used optionally by Prometheus for Omnibus auto host discovery. 2. Can be optionally run on reputable third-party external PaaS Redis solutions. Google Memorystore and AWS Elasticache are known to work. 3. Can be optionally run on reputable third-party load balancing services (LB PaaS). AWS ELB is known to work. 4. Should be run on reputable third-party object storage (storage PaaS) for cloud implementations. Google Cloud Storage and AWS S3 are known to work. diff --git a/doc/administration/reference_architectures/5k_users.md b/doc/administration/reference_architectures/5k_users.md index a5526986be1..a1921f50e4e 100644 --- a/doc/administration/reference_architectures/5k_users.md +++ b/doc/administration/reference_architectures/5k_users.md @@ -19,9 +19,10 @@ costly-to-operate environment by using the > - **Supported users (approximate):** 5,000 > - **High Availability:** Yes ([Praefect](#configure-praefect-postgresql) needs a third-party PostgreSQL solution for HA) -> - **Cloud Native Hybrid:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) -> - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS -> - **[Latest 5k weekly performance testing results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)** +> - **Cloud Native Hybrid Alternative:** [Yes](#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) +> - **Performance tested weekly with the [GitLab Performance Tool (GPT)](https://gitlab.com/gitlab-org/quality/performance)**: +> - **Test requests per second (RPS) rates:** API: 100 RPS, Web: 10 RPS, Git (Pull): 10 RPS, Git (Push): 2 RPS +> - **[Latest Results](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k)** | Service | Nodes | Configuration | GCP | AWS | Azure | |--------------------------------------------|-------------|-------------------------|-----------------|--------------|----------| @@ -288,7 +289,7 @@ The basic ports to be used are shown in the table below. | 443 | 443 | TCP or HTTPS (*1*) (*2*) | | 22 | 22 | TCP | -- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals) support requires +- (*1*): [Web terminal](../../ci/environments/index.md#web-terminals-deprecated) support requires your load balancer to correctly handle WebSocket connections. When using HTTP or HTTPS proxying, this means your load balancer must be configured to pass through the `Connection` and `Upgrade` hop-by-hop headers. See the @@ -2033,7 +2034,7 @@ on what features you intend to use: | [Mattermost](https://docs.mattermost.com/administration/config-settings.html#file-storage)| No | | [Packages](../packages/index.md#using-object-storage) (optional feature) | Yes | | [Dependency Proxy](../packages/dependency_proxy.md#using-object-storage) (optional feature) | Yes | -| [Pseudonymizer](../pseudonymizer.md#configuration) (optional feature) **(ULTIMATE SELF)** | No | +| [Pseudonymizer](../pseudonymizer.md) (optional feature) | No | | [Autoscale runner caching](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) (optional for improved performance) | No | | [Terraform state files](../terraform_state.md#using-object-storage) | Yes | diff --git a/doc/administration/repository_storage_paths.md b/doc/administration/repository_storage_paths.md index ed50d0e7263..00aa8c214c5 100644 --- a/doc/administration/repository_storage_paths.md +++ b/doc/administration/repository_storage_paths.md @@ -14,7 +14,7 @@ storage is either: - A `path`, which points directly to the directory where the repositories are stored. GitLab directly accessing a directory containing repositories [is deprecated](https://gitlab.com/gitlab-org/gitaly/-/issues/1690). - GitLab should be configured to access GitLab repositories though a `gitaly_address`. + GitLab should be configured to access GitLab repositories through a `gitaly_address`. GitLab allows you to define multiple repository storages to distribute the storage load between several mount points. For example: diff --git a/doc/administration/repository_storage_types.md b/doc/administration/repository_storage_types.md index d2bab9a1e04..a85f678fe95 100644 --- a/doc/administration/repository_storage_types.md +++ b/doc/administration/repository_storage_types.md @@ -101,8 +101,12 @@ To look up a project's hash path using a Rails console: #### From hashed path to project name -Administrators can look up a project's name from its hashed storage path using a Rails console. To -look up a project's name from its hashed storage path: +Administrators can look up a project's name from its hashed storage path using: + +- A Rails console. +- The `config` file in the `*.git` directory. + +To look up a project's name using the Rails console: 1. Start a [Rails console](operations/rails_console.md#starting-a-rails-console-session). 1. Run a command similar to this example: @@ -121,6 +125,14 @@ The output includes the project ID and the project name. For example: => #<Project id:16 it/supportteam/ticketsystem> ``` +To look up a project's name using the `config` file in the `*.git` directory: + +1. Navigate to the to the `*.git` directory. This directory is located in `/var/opt/gitlab/git-data/repositories/@hashed/`, where the first four + characters of the hash are the first two directories in the path under `@hashed/`. For example, on a default Omnibus GitLab installation the + `*.git` directory of the hash `b17eb17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9` would be + `/var/opt/gitlab/git-data/repositories/@hashed/b1/7e/b17ef6d19c7a5b1ee83b907c595526dcb1eb06db8227d650d5dda0a9f4ce8cd9.git`. +1. Open the `config` file and locate the `fullpath=` key under `[gitlab]`. + ### Hashed object pools > [Introduced](https://gitlab.com/gitlab-org/gitaly/-/issues/1606) in GitLab 12.1. diff --git a/doc/administration/restart_gitlab.md b/doc/administration/restart_gitlab.md index b8f09b00773..9579d413bf8 100644 --- a/doc/administration/restart_gitlab.md +++ b/doc/administration/restart_gitlab.md @@ -103,39 +103,15 @@ If you have followed the official installation guide to [install GitLab from source](../install/installation.md), run the following command to restart GitLab: ```shell -sudo service gitlab restart -``` +# For systems running systemd +sudo systemctl restart gitlab.target -The output should be similar to this: - -```plaintext -Shutting down GitLab Puma -Shutting down GitLab Sidekiq -Shutting down GitLab Workhorse -Shutting down GitLab MailRoom -... -GitLab is not running. -Starting GitLab Puma -Starting GitLab Sidekiq -Starting GitLab Workhorse -Starting GitLab MailRoom -... -The GitLab Puma web server with pid 28059 is running. -The GitLab Sidekiq job dispatcher with pid 28176 is running. -The GitLab Workhorse with pid 28122 is running. -The GitLab MailRoom email processor with pid 28114 is running. -GitLab and all its components are up and running. +# For systems running SysV init +sudo service gitlab restart ``` This should restart Puma, Sidekiq, GitLab Workhorse, and [Mailroom](reply_by_email.md) -(if enabled). The init service file that does all the magic can be found on -your server in `/etc/init.d/gitlab`. - ---- - -If you are using other init systems, like `systemd`, you can check the -[GitLab Recipes](https://gitlab.com/gitlab-org/gitlab-recipes/tree/master/init) repository for some unofficial services. These are -**not** officially supported so use them at your own risk. +(if enabled). ## Helm chart installations diff --git a/doc/administration/sidekiq.md b/doc/administration/sidekiq.md index 4aee88ed9cb..8f1119f6868 100644 --- a/doc/administration/sidekiq.md +++ b/doc/administration/sidekiq.md @@ -74,6 +74,20 @@ you want using steps 1 and 2 from the GitLab downloads page. postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 10.10.1.30/32 10.10.1.31/32 10.10.1.32/32 10.10.1.33/32 10.10.1.38/32) ``` +1. If you run multiple Sidekiq nodes with a shared file storage, such as NFS, you must + specify the UIDs and GIDs to ensure they match between servers. Specifying the UIDs + and GIDs prevents permissions issues in the file system. This advice is similar to the + [advice for Geo setups](geo/replication/multiple_servers.md#step-4-configure-the-frontend-application-nodes-on-the-geo-secondary-site): + + ```ruby + user['uid'] = 9000 + user['gid'] = 9000 + web_server['uid'] = 9001 + web_server['gid'] = 9001 + registry['uid'] = 9002 + registry['gid'] = 9002 + ``` + 1. Disable other services: ```ruby diff --git a/doc/administration/troubleshooting/debug.md b/doc/administration/troubleshooting/debug.md index e00243aca0a..744e12d4da1 100644 --- a/doc/administration/troubleshooting/debug.md +++ b/doc/administration/troubleshooting/debug.md @@ -258,7 +258,7 @@ separate Rails process to debug the issue: ### GitLab: API is not accessible This often occurs when GitLab Shell attempts to request authorization via the -[internal API](../../development/internal_api.md) (for example, `http://localhost:8080/api/v4/internal/allowed`), and +[internal API](../../development/internal_api/index.md) (for example, `http://localhost:8080/api/v4/internal/allowed`), and something in the check fails. There are many reasons why this may happen: 1. Timeout connecting to a database (for example, PostgreSQL or Redis) @@ -275,7 +275,7 @@ strace -ttTfyyy -s 1024 -p <PID of puma worker> -o /tmp/puma.txt If you cannot isolate which Unicorn worker is the issue, try to run `strace` on all the Unicorn workers to see where the -[`/internal/allowed`](../../development/internal_api.md) endpoint gets stuck: +[`/internal/allowed`](../../development/internal_api/index.md) endpoint gets stuck: ```shell ps auwx | grep puma | awk '{ print " -p " $2}' | xargs strace -ttTfyyy -s 1024 -o /tmp/puma.txt diff --git a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md index 109f451be5a..87f514a2fdd 100644 --- a/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md +++ b/doc/administration/troubleshooting/gitlab_rails_cheat_sheet.md @@ -965,6 +965,22 @@ license.save License.current # check to make sure it applied ``` +This is needed for example in a known edge-case with +[expired license and multiple LDAP servers](../auth/ldap/ldap-troubleshooting.md#expired-license-causes-errors-with-multiple-ldap-servers). + +### Remove licenses + +To clean up the [License History table](../../user/admin_area/license.md#license-history): + +```ruby +TYPE = :trial? +# or :expired? + +License.select(&TYPE).each(&:destroy!) + +# or even License.all.each(&:destroy!) +``` + ## Registry ### Registry Disk Space Usage by Project diff --git a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md index abeba514e4b..64a6979c016 100644 --- a/doc/administration/troubleshooting/kubernetes_cheat_sheet.md +++ b/doc/administration/troubleshooting/kubernetes_cheat_sheet.md @@ -165,8 +165,16 @@ and they will assist you with any issues you are having. - How to connect to a GitLab PostgreSQL database: + In GitLab 14.2 (chart 5.2) and later: + + ```shell + kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password --database main + ``` + + In GitLab 14.1 (chart 5.1) and earlier: + ```shell - kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole -p + kubectl exec -it <task-runner-pod-name> -- /srv/gitlab/bin/rails dbconsole --include-password ``` - How to get information about Helm installation status: diff --git a/doc/administration/troubleshooting/postgresql.md b/doc/administration/troubleshooting/postgresql.md index 3df957bacf9..f8cb45dd00d 100644 --- a/doc/administration/troubleshooting/postgresql.md +++ b/doc/administration/troubleshooting/postgresql.md @@ -2,10 +2,9 @@ stage: Enablement group: Database info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- -# PostgreSQL +# PostgreSQL **(FREE SELF)** This page contains information about PostgreSQL the GitLab Support team uses when troubleshooting. GitLab makes this information public, so that anyone can diff --git a/doc/api/admin_sidekiq_queues.md b/doc/api/admin_sidekiq_queues.md index 569dfd4c413..079ab96c938 100644 --- a/doc/api/admin_sidekiq_queues.md +++ b/doc/api/admin_sidekiq_queues.md @@ -35,7 +35,7 @@ DELETE /admin/sidekiq/queues/:queue_name | `root_namespace` | string | no | The root namespace of the project | | `subscription_plan` | string | no | The subscription plan of the root namespace (GitLab.com only) | | `caller_id` | string | no | The endpoint or background job that schedule the job (for example: `ProjectsController#create`, `/api/:version/projects/:id`, `PostReceive`) | -| `feature_category` | string | no | The feature category of the background job (for example: `issue_tracking` or `code_review`) | +| `feature_category` | string | no | The feature category of the background job (for example: `team_planning` or `code_review`) | | `worker_class` | string | no | The class of the background job worker (for example: `PostReceive` or `MergeWorker`) | At least one attribute, other than `queue_name`, is required. diff --git a/doc/api/api_resources.md b/doc/api/api_resources.md index 7dc3fd1db21..d496ecbca5b 100644 --- a/doc/api/api_resources.md +++ b/doc/api/api_resources.md @@ -71,6 +71,7 @@ The following API resources are available in the project context: | [Project milestones](milestones.md) | `/projects/:id/milestones` | | [Project snippets](project_snippets.md) | `/projects/:id/snippets` | | [Project templates](project_templates.md) | `/projects/:id/templates` | +| [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/templates` | | [Protected environments](protected_environments.md) | `/projects/:id/protected_environments` | | [Protected branches](protected_branches.md) | `/projects/:id/protected_branches` | | [Protected tags](protected_tags.md) | `/projects/:id/protected_tags` | @@ -88,8 +89,7 @@ The following API resources are available in the project context: | [User-starred metrics dashboards](metrics_user_starred_dashboards.md ) | `/projects/:id/metrics/user_starred_dashboards` | | [Visual Review discussions](visual_review_discussions.md) **(PREMIUM)** | `/projects/:id/merge_requests/:merge_request_id/visual_review_discussions` | | [Vulnerabilities](vulnerabilities.md) **(ULTIMATE)** | `/vulnerabilities/:id` | -| [Vulnerability exports](vulnerability_exports.md) **(ULTIMATE)** | `/projects/:id/vulnerability_exports` | -| [Project vulnerabilities](project_vulnerabilities.md) **(ULTIMATE)** | `/projects/:id/vulnerabilities` | +| [Vulnerability exports](vulnerability_exports.md) **(ULTIMATE)** | `/projects/:id/vulnerability_exports` | | [Vulnerability findings](vulnerability_findings.md) **(ULTIMATE)** | `/projects/:id/vulnerability_findings` | | [Project wikis](wikis.md) | `/projects/:id/wikis` | @@ -171,6 +171,7 @@ The following API resources are available outside of project and group contexts | [Suggestions](suggestions.md) | `/suggestions` | | [System hooks](system_hooks.md) | `/hooks` | | [To-dos](todos.md) | `/todos` | +| [Topics](topics.md) | `/topics` | | [Service Data](usage_data.md) | `/usage_data` (For GitLab instance [Administrator](../user/permissions.md) users only) | | [Users](users.md) | `/users` | | [Validate `.gitlab-ci.yml` file](lint.md) | `/lint` | diff --git a/doc/api/dependencies.md b/doc/api/dependencies.md index b421a32b88a..1ecb78aa26d 100644 --- a/doc/api/dependencies.md +++ b/doc/api/dependencies.md @@ -34,7 +34,7 @@ GET /projects/:id/dependencies?package_manager=yarn,bundler | Attribute | Type | Required | Description | | ------------- | -------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding). | -| `package_manager` | string array | no | Returns dependencies belonging to specified package manager. Valid values: `bundler`, `composer`, `conan`, `go`, `maven`, `npm`, `nuget`, `pip`, `yarn`, or `sbt`. | +| `package_manager` | string array | no | Returns dependencies belonging to specified package manager. Valid values: `bundler`, `composer`, `conan`, `go`, `gradle`, `maven`, `npm`, `nuget`, `pip`, `pipenv`, `yarn`, `sbt`, or `setuptools`. | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/4/dependencies" diff --git a/doc/api/deploy_keys.md b/doc/api/deploy_keys.md index bb719b5bc79..b244384bd6a 100644 --- a/doc/api/deploy_keys.md +++ b/doc/api/deploy_keys.md @@ -15,8 +15,16 @@ endpoint requires an administrator role and is not available on GitLab.com. GET /deploy_keys ``` +Supported attributes: + +| Attribute | Type | Required | Description | +|:------------|:---------|:---------|:----------------------| +| `public` | boolean | **{dotted-circle}** No | Only return deploy keys that are public. Defaults to `false`. | + +Example request: + ```shell -curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/deploy_keys" +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/deploy_keys?public=true" ``` Example response: @@ -27,13 +35,36 @@ Example response: "id": 1, "title": "Public key", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", - "created_at": "2013-10-02T10:12:29Z" + "fingerprint": "7f:72:08:7d:0e:47:48:ec:37:79:b2:76:68:b5:87:65", + "created_at": "2013-10-02T10:12:29Z", + "projects_with_write_access": [ + { + "id": 73, + "description": null, + "name": "project2", + "name_with_namespace": "Sidney Jones / project2", + "path": "project2", + "path_with_namespace": "sidney_jones/project2", + "created_at": "2021-10-25T18:33:17.550Z" + }, + { + "id": 74, + "description": null, + "name": "project3", + "name_with_namespace": "Sidney Jones / project3", + "path": "project3", + "path_with_namespace": "sidney_jones/project3", + "created_at": "2021-10-25T18:33:17.666Z" + } + ] }, { "id": 3, "title": "Another Public key", "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAIEAiPWx6WM4lhHNedGfBpPJNPpZ7yKu+dnn1SJejgt4596k6YjzGGphH2TUxwKzxcKDKKezwkpfnxPkSMkuEspGRt/aZZ9wa++Oi7Qkr8prgHc4soW6NUlfDzpvZK2H5E7eQaSeP3SAwGmQKUFHCddNaP0L+hM7zhFNzjFvpaMgJw0=", - "created_at": "2013-10-02T11:12:29Z" + "fingerprint": "64:d3:73:d4:83:70:ab:41:96:68:d5:3d:a5:b0:34:ea", + "created_at": "2013-10-02T11:12:29Z", + "projects_with_write_access": [] } ] ``` diff --git a/doc/api/deployments.md b/doc/api/deployments.md index 5f710271d60..253bc76737b 100644 --- a/doc/api/deployments.md +++ b/doc/api/deployments.md @@ -358,6 +358,12 @@ Example response: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35739) in GitLab 12.7. +NOTE: +Not all deployments can be associated with merge requests. +Please see +[Track what merge requests were deployed to an environment](../ci/environments/index.md#track-newly-included-merge-requests-per-deployment) +for more information. + This API retrieves the list of merge requests shipped with a given deployment: ```plaintext diff --git a/doc/api/dora/metrics.md b/doc/api/dora/metrics.md index 4fbd2b0fa80..3e7b7800034 100644 --- a/doc/api/dora/metrics.md +++ b/doc/api/dora/metrics.md @@ -10,7 +10,7 @@ type: reference, api > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/279039) in GitLab 13.10. > - The legacy key/value pair `{ "<date>" => "<value>" }` was removed from the payload in GitLab 14.0. -All methods require [reporter permissions and above](../../user/permissions.md). +All methods require at least the Reporter [role](../../user/permissions.md). ## Get project-level DORA metrics diff --git a/doc/api/events.md b/doc/api/events.md index 2d173f0053f..265fc0e5fd2 100644 --- a/doc/api/events.md +++ b/doc/api/events.md @@ -8,55 +8,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w ## Filter parameters -### Action Types - -Available types for the `action` parameter, and the resources that might be affected: - -- `approved` - - Merge request -- `closed` - - Epic **(PREMIUM)** - - Issue - - Merge request - - Milestone -- `commented` on any `Noteable` record. - - Alert - - Commit - - Design - - Issue - - Merge request - - Snippet -- `created` - - Design - - Epic **(PREMIUM)** - - Issue - - Merge request - - Milestone - - Project - - Wiki page -- `destroyed` - - Design - - Milestone - - Wiki page -- `expired` - - Project membership -- `joined` - - Project membership -- `left` - - Project membership -- `merged` - - Merge request -- `pushed` commits to (or deleted commits from) a repository, individually or in bulk. - - Project -- `reopened` - - Epic **(PREMIUM)** - - Issue - - Merge request - - Milestone -- `updated` - - Design - - Wiki page +### Actions +See [User contribution events](../user/index.md#user-contribution-events) for available types for the `action` parameter. These options are in lowercase. ### Target Types @@ -72,6 +26,7 @@ Available target types for the `target_type` parameter are: - `user` These options are in lowercase. +Events associated with epics are not available using the API. ### Date formatting @@ -88,6 +43,7 @@ GitLab removes events older than 3 years from the events table for performance r ## List currently authenticated user's events Get a list of events for the authenticated user. Scope `read_user` or `api` is required. +Events associated with epics are not available using the API. ```plaintext GET /events @@ -97,7 +53,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | -| `action` | string | no | Include only events of a particular [action type](#action-types) | +| `action` | string | no | Include only events of a particular [action type](#actions) | | `target_type` | string | no | Include only events of a particular [target type](#target-types) | | `before` | date | no | Include only events created before a particular date. [View how to format dates](#date-formatting). | | `after` | date | no | Include only events created after a particular date. [View how to format dates](#date-formatting). | @@ -160,6 +116,7 @@ Example response: ### Get user contribution events Get the contribution events for the specified user, sorted from newest to oldest. Scope `read_user` or `api` is required. +Events associated with epics are not available using API. ```plaintext GET /users/:id/events @@ -170,7 +127,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `id` | integer | yes | The ID or Username of the user | -| `action` | string | no | Include only events of a particular [action type](#action-types) | +| `action` | string | no | Include only events of a particular [action type](#actions) | | `target_type` | string | no | Include only events of a particular [target type](#target-types) | | `before` | date | no | Include only events created before a particular date. [View how to format dates](#date-formatting). | | `after` | date | no | Include only events created after a particular date. [View how to format dates](#date-formatting). | @@ -308,7 +265,7 @@ Parameters: | Attribute | Type | Required | Description | | --------- | ---- | -------- | ----------- | | `project_id` | integer/string | yes | The ID or [URL-encoded path of the project](index.md#namespaced-path-encoding) | -| `action` | string | no | Include only events of a particular [action type](#action-types) | +| `action` | string | no | Include only events of a particular [action type](#actions) | | `target_type` | string | no | Include only events of a particular [target type](#target-types) | | `before` | date | no | Include only events created before a particular date. [View how to format dates](#date-formatting). | | `after` | date | no | Include only events created after a particular date. [View how to format dates](#date-formatting). | diff --git a/doc/api/geo_nodes.md b/doc/api/geo_nodes.md index d9b23485fd5..fb821824dd1 100644 --- a/doc/api/geo_nodes.md +++ b/doc/api/geo_nodes.md @@ -306,11 +306,6 @@ Example response: "health": "Healthy", "health_status": "Healthy", "missing_oauth_application": false, - "attachments_count": 1, - "attachments_synced_count": null, - "attachments_failed_count": null, - "attachments_synced_missing_on_primary_count": 0, - "attachments_synced_in_percentage": "0.00%", "db_replication_lag_seconds": null, "lfs_objects_count": 0, "lfs_objects_synced_count": null, @@ -465,11 +460,6 @@ Example response: "health": "Healthy", "health_status": "Healthy", "missing_oauth_application": false, - "attachments_count": 1, - "attachments_synced_count": 1, - "attachments_failed_count": 0, - "attachments_synced_missing_on_primary_count": 0, - "attachments_synced_in_percentage": "100.00%", "db_replication_lag_seconds": 0, "lfs_objects_count": 0, "lfs_objects_synced_count": 0, @@ -628,11 +618,6 @@ Example response: "health": "Healthy", "health_status": "Healthy", "missing_oauth_application": false, - "attachments_count": 1, - "attachments_synced_count": 1, - "attachments_failed_count": 0, - "attachments_synced_missing_on_primary_count": 0, - "attachments_synced_in_percentage": "100.00%", "db_replication_lag_seconds": 0, "lfs_objects_count": 0, "lfs_objects_synced_count": 0, diff --git a/doc/api/graphql/reference/index.md b/doc/api/graphql/reference/index.md index e0f18f931f5..34af9736056 100644 --- a/doc/api/graphql/reference/index.md +++ b/doc/api/graphql/reference/index.md @@ -419,6 +419,22 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="querytimelogsstarttime"></a>`startTime` | [`Time`](#time) | List timelogs within a time range where the logged time is equal to or after startTime. | | <a id="querytimelogsusername"></a>`username` | [`String`](#string) | List timelogs for a user. | +### `Query.topics` + +Find project topics. + +Returns [`TopicConnection`](#topicconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="querytopicssearch"></a>`search` | [`String`](#string) | Search query for topic name. | + ### `Query.usageTrendsMeasurements` Get statistics on the instance. @@ -484,6 +500,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="queryvulnerabilitiesclusterid"></a>`clusterId` | [`[ClustersClusterID!]`](#clustersclusterid) | Filter vulnerabilities by `cluster_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="queryvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | | <a id="queryvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | | <a id="queryvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | @@ -1009,6 +1026,31 @@ Input type: `ConfigureSastInput` | <a id="mutationconfiguresasterrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationconfiguresastsuccesspath"></a>`successPath` | [`String`](#string) | Redirect path to use when the response is successful. | +### `Mutation.configureSastIac` + +Enable SAST IaC for a project in a new or +modified `.gitlab-ci.yml` file in a new branch. The new +branch and a URL to create a merge request are a part of the +response. + +Input type: `ConfigureSastIacInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationconfiguresastiacclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationconfiguresastiacprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationconfiguresastiacbranch"></a>`branch` | [`String`](#string) | Branch that has the new/modified `.gitlab-ci.yml` file. | +| <a id="mutationconfiguresastiacclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationconfiguresastiacerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationconfiguresastiacsuccesspath"></a>`successPath` | [`String`](#string) | Redirect path to use when the response is successful. | + ### `Mutation.configureSecretDetection` Configure Secret Detection for a project by enabling Secret Detection @@ -1034,6 +1076,27 @@ Input type: `ConfigureSecretDetectionInput` | <a id="mutationconfiguresecretdetectionerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationconfiguresecretdetectionsuccesspath"></a>`successPath` | [`String`](#string) | Redirect path to use when the response is successful. | +### `Mutation.corpusCreate` + +Available only when feature flag `corpus_management` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. + +Input type: `CorpusCreateInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationcorpuscreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationcorpuscreatefullpath"></a>`fullPath` | [`ID!`](#id) | Project the corpus belongs to. | +| <a id="mutationcorpuscreatepackageid"></a>`packageId` | [`PackagesPackageID!`](#packagespackageid) | ID of the corpus package. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationcorpuscreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationcorpuscreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.createAlertIssue` Input type: `CreateAlertIssueInput` @@ -1554,7 +1617,7 @@ Input type: `DastProfileCreateInput` | ---- | ---- | ----------- | | <a id="mutationdastprofilecreatebranchname"></a>`branchName` | [`String`](#string) | Associated branch. | | <a id="mutationdastprofilecreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdastprofilecreatedastprofileschedule"></a>`dastProfileSchedule` | [`DastProfileScheduleInput`](#dastprofilescheduleinput) | Represents a DAST Profile Schedule. Results in an error if `dast_on_demand_scans_scheduler` feature flag is disabled. | +| <a id="mutationdastprofilecreatedastprofileschedule"></a>`dastProfileSchedule` | [`DastProfileScheduleInput`](#dastprofilescheduleinput) | Represents a DAST Profile Schedule. | | <a id="mutationdastprofilecreatedastscannerprofileid"></a>`dastScannerProfileId` | [`DastScannerProfileID!`](#dastscannerprofileid) | ID of the scanner profile to be associated. | | <a id="mutationdastprofilecreatedastsiteprofileid"></a>`dastSiteProfileId` | [`DastSiteProfileID!`](#dastsiteprofileid) | ID of the site profile to be associated. | | <a id="mutationdastprofilecreatedescription"></a>`description` | [`String`](#string) | Description of the profile. Defaults to an empty string. | @@ -1598,7 +1661,7 @@ Input type: `DastProfileRunInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationdastprofilerunclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdastprofilerunfullpath"></a>`fullPath` | [`ID!`](#id) | Full path for the project the scanner profile belongs to. | +| <a id="mutationdastprofilerunfullpath"></a>`fullPath` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Full path not required to qualify Global ID. Deprecated in 14.5. | | <a id="mutationdastprofilerunid"></a>`id` | [`DastProfileID!`](#dastprofileid) | ID of the profile to be used for the scan. | #### Fields @@ -1619,11 +1682,11 @@ Input type: `DastProfileUpdateInput` | ---- | ---- | ----------- | | <a id="mutationdastprofileupdatebranchname"></a>`branchName` | [`String`](#string) | Associated branch. | | <a id="mutationdastprofileupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdastprofileupdatedastprofileschedule"></a>`dastProfileSchedule` | [`DastProfileScheduleInput`](#dastprofilescheduleinput) | Represents a DAST profile schedule. Results in an error if `dast_on_demand_scans_scheduler` feature flag is disabled. | +| <a id="mutationdastprofileupdatedastprofileschedule"></a>`dastProfileSchedule` | [`DastProfileScheduleInput`](#dastprofilescheduleinput) | Represents a DAST profile schedule. | | <a id="mutationdastprofileupdatedastscannerprofileid"></a>`dastScannerProfileId` | [`DastScannerProfileID`](#dastscannerprofileid) | ID of the scanner profile to be associated. | | <a id="mutationdastprofileupdatedastsiteprofileid"></a>`dastSiteProfileId` | [`DastSiteProfileID`](#dastsiteprofileid) | ID of the site profile to be associated. | | <a id="mutationdastprofileupdatedescription"></a>`description` | [`String`](#string) | Description of the profile. Defaults to an empty string. | -| <a id="mutationdastprofileupdatefullpath"></a>`fullPath` | [`ID!`](#id) | Project the profile belongs to. | +| <a id="mutationdastprofileupdatefullpath"></a>`fullPath` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Full path not required to qualify Global ID. Deprecated in 14.5. | | <a id="mutationdastprofileupdateid"></a>`id` | [`DastProfileID!`](#dastprofileid) | ID of the profile to be deleted. | | <a id="mutationdastprofileupdatename"></a>`name` | [`String`](#string) | Name of the profile. | | <a id="mutationdastprofileupdaterunafterupdate"></a>`runAfterUpdate` | [`Boolean`](#boolean) | Run scan using profile after update. Defaults to false. | @@ -1671,7 +1734,7 @@ Input type: `DastScannerProfileDeleteInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationdastscannerprofiledeleteclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdastscannerprofiledeletefullpath"></a>`fullPath` | [`ID!`](#id) | Full path for the project the scanner profile belongs to. | +| <a id="mutationdastscannerprofiledeletefullpath"></a>`fullPath` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Full path not required to qualify Global ID. Deprecated in 14.5. | | <a id="mutationdastscannerprofiledeleteid"></a>`id` | [`DastScannerProfileID!`](#dastscannerprofileid) | ID of the scanner profile to be deleted. | #### Fields @@ -1690,7 +1753,7 @@ Input type: `DastScannerProfileUpdateInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationdastscannerprofileupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdastscannerprofileupdatefullpath"></a>`fullPath` | [`ID!`](#id) | Project the scanner profile belongs to. | +| <a id="mutationdastscannerprofileupdatefullpath"></a>`fullPath` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Full path not required to qualify Global ID. Deprecated in 14.5. | | <a id="mutationdastscannerprofileupdateid"></a>`id` | [`DastScannerProfileID!`](#dastscannerprofileid) | ID of the scanner profile to be updated. | | <a id="mutationdastscannerprofileupdateprofilename"></a>`profileName` | [`String!`](#string) | Name of the scanner profile. | | <a id="mutationdastscannerprofileupdatescantype"></a>`scanType` | [`DastScanTypeEnum`](#dastscantypeenum) | Indicates the type of DAST scan that will run. Either a Passive Scan or an Active Scan. | @@ -1741,7 +1804,7 @@ Input type: `DastSiteProfileDeleteInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationdastsiteprofiledeleteclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationdastsiteprofiledeletefullpath"></a>`fullPath` | [`ID!`](#id) | Project the site profile belongs to. | +| <a id="mutationdastsiteprofiledeletefullpath"></a>`fullPath` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Full path not required to qualify Global ID. Deprecated in 14.5. | | <a id="mutationdastsiteprofiledeleteid"></a>`id` | [`DastSiteProfileID!`](#dastsiteprofileid) | ID of the site profile to be deleted. | #### Fields @@ -1762,7 +1825,7 @@ Input type: `DastSiteProfileUpdateInput` | <a id="mutationdastsiteprofileupdateauth"></a>`auth` | [`DastSiteProfileAuthInput`](#dastsiteprofileauthinput) | Parameters for authentication. | | <a id="mutationdastsiteprofileupdateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationdastsiteprofileupdateexcludedurls"></a>`excludedUrls` | [`[String!]`](#string) | URLs to skip during an authenticated scan. | -| <a id="mutationdastsiteprofileupdatefullpath"></a>`fullPath` | [`ID!`](#id) | Project the site profile belongs to. | +| <a id="mutationdastsiteprofileupdatefullpath"></a>`fullPath` **{warning-solid}** | [`ID`](#id) | **Deprecated:** Full path not required to qualify Global ID. Deprecated in 14.5. | | <a id="mutationdastsiteprofileupdateid"></a>`id` | [`DastSiteProfileID!`](#dastsiteprofileid) | ID of the site profile to be updated. | | <a id="mutationdastsiteprofileupdateprofilename"></a>`profileName` | [`String!`](#string) | Name of the site profile. | | <a id="mutationdastsiteprofileupdaterequestheaders"></a>`requestHeaders` | [`String`](#string) | Comma-separated list of request header names and values to be added to every request made by DAST. | @@ -2774,6 +2837,28 @@ Input type: `IssueSetConfidentialInput` | <a id="mutationissuesetconfidentialerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationissuesetconfidentialissue"></a>`issue` | [`Issue`](#issue) | Issue after mutation. | +### `Mutation.issueSetCrmContacts` + +Input type: `IssueSetCrmContactsInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationissuesetcrmcontactsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationissuesetcrmcontactscrmcontactids"></a>`crmContactIds` | [`[CustomerRelationsContactID!]!`](#customerrelationscontactid) | Customer relations contact IDs to set. Replaces existing contacts by default. | +| <a id="mutationissuesetcrmcontactsiid"></a>`iid` | [`String!`](#string) | IID of the issue to mutate. | +| <a id="mutationissuesetcrmcontactsoperationmode"></a>`operationMode` | [`MutationOperationMode`](#mutationoperationmode) | Changes the operation mode. Defaults to REPLACE. | +| <a id="mutationissuesetcrmcontactsprojectpath"></a>`projectPath` | [`ID!`](#id) | Project the issue to mutate is in. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationissuesetcrmcontactsclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationissuesetcrmcontactserrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationissuesetcrmcontactsissue"></a>`issue` | [`Issue`](#issue) | Issue after mutation. | + ### `Mutation.issueSetDueDate` Input type: `IssueSetDueDateInput` @@ -3401,30 +3486,28 @@ Input type: `MergeRequestSetSubscriptionInput` | <a id="mutationmergerequestsetsubscriptionerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationmergerequestsetsubscriptionmergerequest"></a>`mergeRequest` | [`MergeRequest`](#mergerequest) | Merge request after mutation. | -### `Mutation.mergeRequestSetWip` +### `Mutation.mergeRequestToggleAttentionRequested` -WARNING: -**Deprecated** in 13.12. -Use mergeRequestSetDraft. +Available only when feature flag `mr_attention_requests` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. -Input type: `MergeRequestSetWipInput` +Input type: `MergeRequestToggleAttentionRequestedInput` #### Arguments | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationmergerequestsetwipclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationmergerequestsetwipiid"></a>`iid` | [`String!`](#string) | IID of the merge request to mutate. | -| <a id="mutationmergerequestsetwipprojectpath"></a>`projectPath` | [`ID!`](#id) | Project the merge request to mutate is in. | -| <a id="mutationmergerequestsetwipwip"></a>`wip` | [`Boolean!`](#boolean) | Whether or not to set the merge request as a draft. | +| <a id="mutationmergerequesttoggleattentionrequestedclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationmergerequesttoggleattentionrequestediid"></a>`iid` | [`String!`](#string) | IID of the merge request to mutate. | +| <a id="mutationmergerequesttoggleattentionrequestedprojectpath"></a>`projectPath` | [`ID!`](#id) | Project the merge request to mutate is in. | +| <a id="mutationmergerequesttoggleattentionrequesteduserid"></a>`userId` | [`UserID!`](#userid) | User ID for the user to toggle attention requested. | #### Fields | Name | Type | Description | | ---- | ---- | ----------- | -| <a id="mutationmergerequestsetwipclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | -| <a id="mutationmergerequestsetwiperrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | -| <a id="mutationmergerequestsetwipmergerequest"></a>`mergeRequest` | [`MergeRequest`](#mergerequest) | Merge request after mutation. | +| <a id="mutationmergerequesttoggleattentionrequestedclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationmergerequesttoggleattentionrequestederrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | +| <a id="mutationmergerequesttoggleattentionrequestedmergerequest"></a>`mergeRequest` | [`MergeRequest`](#mergerequest) | Merge request after mutation. | ### `Mutation.mergeRequestUpdate` @@ -4042,6 +4125,7 @@ Input type: `ScanExecutionPolicyCommitInput` | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mutationscanexecutionpolicycommitclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationscanexecutionpolicycommitname"></a>`name` | [`String`](#string) | Name of the policy. If the name is null, the `name` field from `policy_yaml` is used. | | <a id="mutationscanexecutionpolicycommitoperationmode"></a>`operationMode` | [`MutationOperationMode!`](#mutationoperationmode) | Changes the operation mode. | | <a id="mutationscanexecutionpolicycommitpolicyyaml"></a>`policyYaml` | [`String!`](#string) | YAML snippet of the policy. | | <a id="mutationscanexecutionpolicycommitprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | @@ -4096,6 +4180,26 @@ Input type: `SecurityPolicyProjectCreateInput` | <a id="mutationsecuritypolicyprojectcreateerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | | <a id="mutationsecuritypolicyprojectcreateproject"></a>`project` | [`Project`](#project) | Security Policy Project that was created. | +### `Mutation.securityPolicyProjectUnassign` + +Unassigns the security policy project for the given project(`project_path`). + +Input type: `SecurityPolicyProjectUnassignInput` + +#### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationsecuritypolicyprojectunassignclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationsecuritypolicyprojectunassignprojectpath"></a>`projectPath` | [`ID!`](#id) | Full path of the project. | + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="mutationsecuritypolicyprojectunassignclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | +| <a id="mutationsecuritypolicyprojectunassignerrors"></a>`errors` | [`[String!]!`](#string) | Errors encountered during execution of the mutation. | + ### `Mutation.terraformStateDelete` Input type: `TerraformStateDeleteInput` @@ -4710,17 +4814,17 @@ Input type: `VulnerabilityCreateInput` | <a id="mutationvulnerabilitycreateclientmutationid"></a>`clientMutationId` | [`String`](#string) | A unique identifier for the client performing the mutation. | | <a id="mutationvulnerabilitycreateconfidence"></a>`confidence` | [`VulnerabilityConfidence`](#vulnerabilityconfidence) | Confidence of the vulnerability (defaults to `unknown`). | | <a id="mutationvulnerabilitycreateconfirmedat"></a>`confirmedAt` | [`Time`](#time) | Timestamp of when the vulnerability state changed to confirmed (defaults to creation time if status is `confirmed`). | -| <a id="mutationvulnerabilitycreatedescription"></a>`description` | [`String!`](#string) | Description of the vulnerability. | +| <a id="mutationvulnerabilitycreatedescription"></a>`description` | [`String!`](#string) | Long text section that describes the vulnerability in more detail. | | <a id="mutationvulnerabilitycreatedetectedat"></a>`detectedAt` | [`Time`](#time) | Timestamp of when the vulnerability was first detected (defaults to creation time). | | <a id="mutationvulnerabilitycreatedismissedat"></a>`dismissedAt` | [`Time`](#time) | Timestamp of when the vulnerability state changed to dismissed (defaults to creation time if status is `dismissed`). | | <a id="mutationvulnerabilitycreateidentifiers"></a>`identifiers` | [`[VulnerabilityIdentifierInput!]!`](#vulnerabilityidentifierinput) | Array of CVE or CWE identifiers for the vulnerability. | -| <a id="mutationvulnerabilitycreatemessage"></a>`message` | [`String`](#string) | Additional information about the vulnerability. | +| <a id="mutationvulnerabilitycreatemessage"></a>`message` | [`String`](#string) | Short text section that describes the vulnerability. This may include the finding's specific information. | | <a id="mutationvulnerabilitycreatename"></a>`name` | [`String!`](#string) | Name of the vulnerability. | | <a id="mutationvulnerabilitycreateproject"></a>`project` | [`ProjectID!`](#projectid) | ID of the project to attach the vulnerability to. | | <a id="mutationvulnerabilitycreateresolvedat"></a>`resolvedAt` | [`Time`](#time) | Timestamp of when the vulnerability state changed to resolved (defaults to creation time if status is `resolved`). | | <a id="mutationvulnerabilitycreatescanner"></a>`scanner` | [`VulnerabilityScannerInput!`](#vulnerabilityscannerinput) | Information about the scanner used to discover the vulnerability. | | <a id="mutationvulnerabilitycreateseverity"></a>`severity` | [`VulnerabilitySeverity`](#vulnerabilityseverity) | Severity of the vulnerability (defaults to `unknown`). | -| <a id="mutationvulnerabilitycreatesolution"></a>`solution` | [`String`](#string) | How to fix this vulnerability. | +| <a id="mutationvulnerabilitycreatesolution"></a>`solution` | [`String`](#string) | Instructions for how to fix the vulnerability. | | <a id="mutationvulnerabilitycreatestate"></a>`state` | [`VulnerabilityState`](#vulnerabilitystate) | State of the vulnerability (defaults to `detected`). | #### Fields @@ -5563,6 +5667,29 @@ The edge type for [`ContainerRepositoryTag`](#containerrepositorytag). | <a id="containerrepositorytagedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="containerrepositorytagedgenode"></a>`node` | [`ContainerRepositoryTag`](#containerrepositorytag) | The item at the end of the edge. | +#### `CoverageFuzzingCorpusConnection` + +The connection type for [`CoverageFuzzingCorpus`](#coveragefuzzingcorpus). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="coveragefuzzingcorpusconnectionedges"></a>`edges` | [`[CoverageFuzzingCorpusEdge]`](#coveragefuzzingcorpusedge) | A list of edges. | +| <a id="coveragefuzzingcorpusconnectionnodes"></a>`nodes` | [`[CoverageFuzzingCorpus]`](#coveragefuzzingcorpus) | A list of nodes. | +| <a id="coveragefuzzingcorpusconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `CoverageFuzzingCorpusEdge` + +The edge type for [`CoverageFuzzingCorpus`](#coveragefuzzingcorpus). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="coveragefuzzingcorpusedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="coveragefuzzingcorpusedgenode"></a>`node` | [`CoverageFuzzingCorpus`](#coveragefuzzingcorpus) | The item at the end of the edge. | + #### `CustomEmojiConnection` The connection type for [`CustomEmoji`](#customemoji). @@ -7577,6 +7704,29 @@ The edge type for [`Todo`](#todo). | <a id="todoedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | | <a id="todoedgenode"></a>`node` | [`Todo`](#todo) | The item at the end of the edge. | +#### `TopicConnection` + +The connection type for [`Topic`](#topic). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="topicconnectionedges"></a>`edges` | [`[TopicEdge]`](#topicedge) | A list of edges. | +| <a id="topicconnectionnodes"></a>`nodes` | [`[Topic]`](#topic) | A list of nodes. | +| <a id="topicconnectionpageinfo"></a>`pageInfo` | [`PageInfo!`](#pageinfo) | Information to aid in pagination. | + +#### `TopicEdge` + +The edge type for [`Topic`](#topic). + +##### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="topicedgecursor"></a>`cursor` | [`String!`](#string) | A cursor for use in pagination. | +| <a id="topicedgenode"></a>`node` | [`Topic`](#topic) | The item at the end of the edge. | + #### `TreeConnection` The connection type for [`Tree`](#tree). @@ -8535,6 +8685,7 @@ Represents the total number of issues and their weights for a particular day. | ---- | ---- | ----------- | | <a id="cijobartifactdownloadpath"></a>`downloadPath` | [`String`](#string) | URL for downloading the artifact's file. | | <a id="cijobartifactfiletype"></a>`fileType` | [`JobArtifactFileType`](#jobartifactfiletype) | File type of the artifact. | +| <a id="cijobartifactname"></a>`name` | [`String`](#string) | File name of the artifact. | ### `CiJobTokenScopeType` @@ -8553,6 +8704,7 @@ Represents the total number of issues and their weights for a particular day. | <a id="ciminutesnamespacemonthlyusageminutes"></a>`minutes` | [`Int`](#int) | Total number of minutes used by all projects in the namespace. | | <a id="ciminutesnamespacemonthlyusagemonth"></a>`month` | [`String`](#string) | Month related to the usage data. | | <a id="ciminutesnamespacemonthlyusageprojects"></a>`projects` | [`CiMinutesProjectMonthlyUsageConnection`](#ciminutesprojectmonthlyusageconnection) | CI minutes usage data for projects in the namespace. (see [Connections](#connections)) | +| <a id="ciminutesnamespacemonthlyusagesharedrunnersduration"></a>`sharedRunnersDuration` | [`Int`](#int) | Total numbers of minutes used by the shared runners in the namespace. | ### `CiMinutesProjectMonthlyUsage` @@ -8696,6 +8848,8 @@ Represents a code quality degradation on the pipeline. | <a id="commitauthoreddate"></a>`authoredDate` | [`Time`](#time) | Timestamp of when the commit was authored. | | <a id="commitdescription"></a>`description` | [`String`](#string) | Description of the commit message. | | <a id="commitdescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | +| <a id="commitfulltitle"></a>`fullTitle` | [`String`](#string) | Full title of the commit message. | +| <a id="commitfulltitlehtml"></a>`fullTitleHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `full_title`. | | <a id="commitid"></a>`id` | [`ID!`](#id) | ID (global ID) of the commit. | | <a id="commitmessage"></a>`message` | [`String`](#string) | Raw commit message. | | <a id="commitsha"></a>`sha` | [`String!`](#string) | SHA1 ID of the commit. | @@ -8723,6 +8877,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="commitpipelinesref"></a>`ref` | [`String`](#string) | Filter pipelines by the ref they are run for. | +| <a id="commitpipelinesscope"></a>`scope` | [`PipelineScopeEnum`](#pipelinescopeenum) | Filter pipelines by scope. | | <a id="commitpipelinessha"></a>`sha` | [`String`](#string) | Filter pipelines by the sha of the commit they are run for. | | <a id="commitpipelinessource"></a>`source` | [`String`](#string) | Filter pipelines by their source. Will be ignored if `dast_view_scans` feature flag is disabled. | | <a id="commitpipelinesstatus"></a>`status` | [`PipelineStatusEnum`](#pipelinestatusenum) | Filter pipelines by their status. | @@ -8875,6 +9030,17 @@ A tag from a container repository. | <a id="containerrepositorytagshortrevision"></a>`shortRevision` | [`String`](#string) | Short revision of the tag. | | <a id="containerrepositorytagtotalsize"></a>`totalSize` | [`BigInt`](#bigint) | Size of the tag. | +### `CoverageFuzzingCorpus` + +Corpus for a coverage fuzzing job. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="coveragefuzzingcorpusid"></a>`id` | [`AppSecFuzzingCoverageCorpusID!`](#appsecfuzzingcoveragecorpusid) | ID of the corpus. | +| <a id="coveragefuzzingcorpuspackage"></a>`package` | [`PackageDetailsType!`](#packagedetailstype) | Package of the corpus. | + ### `CurrentLicense` Represents the current license. @@ -8887,6 +9053,7 @@ Represents the current license. | <a id="currentlicensebillableuserscount"></a>`billableUsersCount` | [`Int`](#int) | Number of billable users on the system. | | <a id="currentlicenseblockchangesat"></a>`blockChangesAt` | [`Date`](#date) | Date, including grace period, when licensed features will be blocked. | | <a id="currentlicensecompany"></a>`company` | [`String`](#string) | Company of the licensee. | +| <a id="currentlicensecreatedat"></a>`createdAt` | [`Date`](#date) | Date when the license was added. | | <a id="currentlicenseemail"></a>`email` | [`String`](#string) | Email of the licensee. | | <a id="currentlicenseexpiresat"></a>`expiresAt` | [`Date`](#date) | Date when the license expires. | | <a id="currentlicenseid"></a>`id` | [`ID!`](#id) | ID of the license. | @@ -8950,7 +9117,7 @@ Represents a DAST Profile. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="dastprofilebranch"></a>`branch` | [`DastProfileBranch`](#dastprofilebranch) | Associated branch. | -| <a id="dastprofiledastprofileschedule"></a>`dastProfileSchedule` | [`DastProfileSchedule`](#dastprofileschedule) | Associated profile schedule. Will always return `null` if `dast_on_demand_scans_scheduler` feature flag is disabled. | +| <a id="dastprofiledastprofileschedule"></a>`dastProfileSchedule` | [`DastProfileSchedule`](#dastprofileschedule) | Associated profile schedule. | | <a id="dastprofiledastscannerprofile"></a>`dastScannerProfile` | [`DastScannerProfile`](#dastscannerprofile) | Associated scanner profile. | | <a id="dastprofiledastsiteprofile"></a>`dastSiteProfile` | [`DastSiteProfile`](#dastsiteprofile) | Associated site profile. | | <a id="dastprofiledescription"></a>`description` | [`String`](#string) | Description of the scan. | @@ -9120,6 +9287,7 @@ Dependency proxy manifest. | <a id="dependencyproxymanifestcreatedat"></a>`createdAt` | [`Time!`](#time) | Date of creation. | | <a id="dependencyproxymanifestdigest"></a>`digest` | [`String!`](#string) | Digest of the manifest. | | <a id="dependencyproxymanifestfilename"></a>`fileName` | [`String!`](#string) | Name of the manifest. | +| <a id="dependencyproxymanifestid"></a>`id` | [`DependencyProxyManifestID!`](#dependencyproxymanifestid) | ID of the manifest. | | <a id="dependencyproxymanifestimagename"></a>`imageName` | [`String!`](#string) | Name of the image. | | <a id="dependencyproxymanifestsize"></a>`size` | [`String!`](#string) | Size of the manifest file. | | <a id="dependencyproxymanifestupdatedat"></a>`updatedAt` | [`Time!`](#time) | Date of most recent update. | @@ -10226,7 +10394,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="groupemailsdisabled"></a>`emailsDisabled` | [`Boolean`](#boolean) | Indicates if a group has email notifications disabled. | | <a id="groupepicboards"></a>`epicBoards` | [`EpicBoardConnection`](#epicboardconnection) | Find epic boards. (see [Connections](#connections)) | | <a id="groupepicsenabled"></a>`epicsEnabled` | [`Boolean`](#boolean) | Indicates if Epics are enabled for namespace. | -| <a id="groupexternalauditeventdestinations"></a>`externalAuditEventDestinations` | [`ExternalAuditEventDestinationConnection`](#externalauditeventdestinationconnection) | External locations that receive audit events belonging to the group. Available only when feature flag `ff_external_audit_events_namespace` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. (see [Connections](#connections)) | +| <a id="groupexternalauditeventdestinations"></a>`externalAuditEventDestinations` | [`ExternalAuditEventDestinationConnection`](#externalauditeventdestinationconnection) | External locations that receive audit events belonging to the group. (see [Connections](#connections)) | | <a id="groupfullname"></a>`fullName` | [`String!`](#string) | Full name of the namespace. | | <a id="groupfullpath"></a>`fullPath` | [`ID!`](#id) | Full path of the namespace. | | <a id="groupid"></a>`id` | [`ID!`](#id) | ID of the namespace. | @@ -10580,6 +10748,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="groupmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | | <a id="groupmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="groupmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="groupmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="groupmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="groupmergerequestsincludesubgroups"></a>`includeSubgroups` | [`Boolean`](#boolean) | Include merge requests belonging to subgroups. | | <a id="groupmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | @@ -10716,6 +10886,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="groupvulnerabilitiesclusterid"></a>`clusterId` | [`[ClustersClusterID!]`](#clustersclusterid) | Filter vulnerabilities by `cluster_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="groupvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | | <a id="groupvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | | <a id="groupvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | @@ -10840,6 +11011,19 @@ Represents the Geo sync and verification state of a group wiki repository. | <a id="groupwikirepositoryregistryretrycount"></a>`retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the GroupWikiRepositoryRegistry. | | <a id="groupwikirepositoryregistrystate"></a>`state` | [`RegistryState`](#registrystate) | Sync state of the GroupWikiRepositoryRegistry. | +### `HelmFileMetadata` + +Helm file metadata. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="helmfilemetadatachannel"></a>`channel` | [`String!`](#string) | Channel of the Helm chart. | +| <a id="helmfilemetadatacreatedat"></a>`createdAt` | [`Time!`](#time) | Date of creation. | +| <a id="helmfilemetadatametadata"></a>`metadata` | [`PackageHelmMetadataType!`](#packagehelmmetadatatype) | Metadata of the Helm chart. | +| <a id="helmfilemetadataupdatedat"></a>`updatedAt` | [`Time!`](#time) | Date of most recent update. | + ### `IncidentManagementOncallRotation` Describes an incident management on-call rotation. @@ -11258,6 +11442,7 @@ Represents an entry from the Cloud License history. | <a id="licensehistoryentryactivatedat"></a>`activatedAt` | [`Date`](#date) | Date when the license was activated. | | <a id="licensehistoryentryblockchangesat"></a>`blockChangesAt` | [`Date`](#date) | Date, including grace period, when licensed features will be blocked. | | <a id="licensehistoryentrycompany"></a>`company` | [`String`](#string) | Company of the licensee. | +| <a id="licensehistoryentrycreatedat"></a>`createdAt` | [`Date`](#date) | Date when the license was added. | | <a id="licensehistoryentryemail"></a>`email` | [`String`](#string) | Email of the licensee. | | <a id="licensehistoryentryexpiresat"></a>`expiresAt` | [`Date`](#date) | Date when the license expires. | | <a id="licensehistoryentryid"></a>`id` | [`ID!`](#id) | ID of the license. | @@ -11305,7 +11490,7 @@ Maven metadata. | <a id="mergerequestconflicts"></a>`conflicts` | [`Boolean!`](#boolean) | Indicates if the merge request has conflicts. | | <a id="mergerequestcreatedat"></a>`createdAt` | [`Time!`](#time) | Timestamp of when the merge request was created. | | <a id="mergerequestdefaultmergecommitmessage"></a>`defaultMergeCommitMessage` | [`String`](#string) | Default merge commit message of the merge request. | -| <a id="mergerequestdefaultmergecommitmessagewithdescription"></a>`defaultMergeCommitMessageWithDescription` | [`String`](#string) | Default merge commit message of the merge request with description. | +| <a id="mergerequestdefaultmergecommitmessagewithdescription"></a>`defaultMergeCommitMessageWithDescription` **{warning-solid}** | [`String`](#string) | **Deprecated** in 14.5. Define merge commit template in project and use `defaultMergeCommitMessage`. | | <a id="mergerequestdefaultsquashcommitmessage"></a>`defaultSquashCommitMessage` | [`String`](#string) | Default squash commit message of the merge request. | | <a id="mergerequestdescription"></a>`description` | [`String`](#string) | Description of the merge request (Markdown rendered as HTML for caching). | | <a id="mergerequestdescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | @@ -11375,7 +11560,6 @@ Maven metadata. | <a id="mergerequestusernotescount"></a>`userNotesCount` | [`Int`](#int) | User notes count of the merge request. | | <a id="mergerequestuserpermissions"></a>`userPermissions` | [`MergeRequestPermissions!`](#mergerequestpermissions) | Permissions for the current user on the resource. | | <a id="mergerequestweburl"></a>`webUrl` | [`String`](#string) | Web URL of the merge request. | -| <a id="mergerequestworkinprogress"></a>`workInProgress` **{warning-solid}** | [`Boolean!`](#boolean) | **Deprecated** in 13.12. Use `draft`. | #### Fields with arguments @@ -11422,6 +11606,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mergerequestpipelinesref"></a>`ref` | [`String`](#string) | Filter pipelines by the ref they are run for. | +| <a id="mergerequestpipelinesscope"></a>`scope` | [`PipelineScopeEnum`](#pipelinescopeenum) | Filter pipelines by scope. | | <a id="mergerequestpipelinessha"></a>`sha` | [`String`](#string) | Filter pipelines by the sha of the commit they are run for. | | <a id="mergerequestpipelinessource"></a>`source` | [`String`](#string) | Filter pipelines by their source. Will be ignored if `dast_view_scans` feature flag is disabled. | | <a id="mergerequestpipelinesstatus"></a>`status` | [`PipelineStatusEnum`](#pipelinestatusenum) | Filter pipelines by their status. | @@ -11494,6 +11679,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mergerequestassigneeassignedmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="mergerequestassigneeassignedmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="mergerequestassigneeassignedmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="mergerequestassigneeassignedmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="mergerequestassigneeassignedmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="mergerequestassigneeassignedmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -11523,6 +11710,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mergerequestassigneeauthoredmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | +| <a id="mergerequestassigneeauthoredmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="mergerequestassigneeauthoredmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="mergerequestassigneeauthoredmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="mergerequestassigneeauthoredmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="mergerequestassigneeauthoredmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -11570,6 +11759,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="mergerequestassigneereviewrequestedmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | | <a id="mergerequestassigneereviewrequestedmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="mergerequestassigneereviewrequestedmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="mergerequestassigneereviewrequestedmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="mergerequestassigneereviewrequestedmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="mergerequestassigneereviewrequestedmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="mergerequestassigneereviewrequestedmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -11740,6 +11931,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mergerequestreviewerassignedmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="mergerequestreviewerassignedmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="mergerequestreviewerassignedmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="mergerequestreviewerassignedmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="mergerequestreviewerassignedmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="mergerequestreviewerassignedmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -11769,6 +11962,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="mergerequestreviewerauthoredmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | +| <a id="mergerequestreviewerauthoredmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="mergerequestreviewerauthoredmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="mergerequestreviewerauthoredmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="mergerequestreviewerauthoredmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="mergerequestreviewerauthoredmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -11816,6 +12011,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="mergerequestreviewerreviewrequestedmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | | <a id="mergerequestreviewerreviewrequestedmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="mergerequestreviewerreviewrequestedmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="mergerequestreviewerreviewrequestedmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="mergerequestreviewerreviewrequestedmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="mergerequestreviewerreviewrequestedmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="mergerequestreviewerreviewrequestedmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -12297,6 +12494,61 @@ Represents the Geo sync and verification state of a package file. | <a id="packagefileregistryretrycount"></a>`retryCount` | [`Int`](#int) | Number of consecutive failed sync attempts of the PackageFileRegistry. | | <a id="packagefileregistrystate"></a>`state` | [`RegistryState`](#registrystate) | Sync state of the PackageFileRegistry. | +### `PackageHelmDependencyType` + +Represents a Helm dependency. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="packagehelmdependencytypealias"></a>`alias` | [`String`](#string) | Alias of the dependency. | +| <a id="packagehelmdependencytypecondition"></a>`condition` | [`String`](#string) | Condition of the dependency. | +| <a id="packagehelmdependencytypeenabled"></a>`enabled` | [`Boolean`](#boolean) | Indicates the dependency is enabled. | +| <a id="packagehelmdependencytypeimportvalues"></a>`importValues` | [`[JSON!]`](#json) | Import-values of the dependency. | +| <a id="packagehelmdependencytypename"></a>`name` | [`String`](#string) | Name of the dependency. | +| <a id="packagehelmdependencytyperepository"></a>`repository` | [`String`](#string) | Repository of the dependency. | +| <a id="packagehelmdependencytypetags"></a>`tags` | [`[String!]`](#string) | Tags of the dependency. | +| <a id="packagehelmdependencytypeversion"></a>`version` | [`String`](#string) | Version of the dependency. | + +### `PackageHelmMaintainerType` + +Represents a Helm maintainer. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="packagehelmmaintainertypeemail"></a>`email` | [`String`](#string) | Email of the maintainer. | +| <a id="packagehelmmaintainertypename"></a>`name` | [`String`](#string) | Name of the maintainer. | +| <a id="packagehelmmaintainertypeurl"></a>`url` | [`String`](#string) | URL of the maintainer. | + +### `PackageHelmMetadataType` + +Represents the contents of a Helm Chart.yml file. + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="packagehelmmetadatatypeannotations"></a>`annotations` | [`JSON`](#json) | Annotations for the chart. | +| <a id="packagehelmmetadatatypeapiversion"></a>`apiVersion` | [`String!`](#string) | API version of the chart. | +| <a id="packagehelmmetadatatypeappversion"></a>`appVersion` | [`String`](#string) | App version of the chart. | +| <a id="packagehelmmetadatatypecondition"></a>`condition` | [`String`](#string) | Condition for the chart. | +| <a id="packagehelmmetadatatypedependencies"></a>`dependencies` | [`[PackageHelmDependencyType!]`](#packagehelmdependencytype) | Dependencies of the chart. | +| <a id="packagehelmmetadatatypedeprecated"></a>`deprecated` | [`Boolean`](#boolean) | Indicates if the chart is deprecated. | +| <a id="packagehelmmetadatatypedescription"></a>`description` | [`String`](#string) | Description of the chart. | +| <a id="packagehelmmetadatatypehome"></a>`home` | [`String`](#string) | URL of the home page. | +| <a id="packagehelmmetadatatypeicon"></a>`icon` | [`String`](#string) | URL to an SVG or PNG image for the chart. | +| <a id="packagehelmmetadatatypekeywords"></a>`keywords` | [`[String!]`](#string) | Keywords for the chart. | +| <a id="packagehelmmetadatatypekubeversion"></a>`kubeVersion` | [`String`](#string) | Kubernetes versions for the chart. | +| <a id="packagehelmmetadatatypemaintainers"></a>`maintainers` | [`[PackageHelmMaintainerType!]`](#packagehelmmaintainertype) | Maintainers of the chart. | +| <a id="packagehelmmetadatatypename"></a>`name` | [`String!`](#string) | Name of the chart. | +| <a id="packagehelmmetadatatypesources"></a>`sources` | [`[String!]`](#string) | URLs of the source code for the chart. | +| <a id="packagehelmmetadatatypetags"></a>`tags` | [`String`](#string) | Tags for the chart. | +| <a id="packagehelmmetadatatypetype"></a>`type` | [`String`](#string) | Type of the chart. | +| <a id="packagehelmmetadatatypeversion"></a>`version` | [`String!`](#string) | Version of the chart. | + ### `PackageSettings` Namespace-level Package Registry settings. @@ -12375,6 +12627,7 @@ Represents a file or directory in the project repository that has been locked. | <a id="pipelinebeforesha"></a>`beforeSha` | [`String`](#string) | Base SHA of the source branch. | | <a id="pipelinecancelable"></a>`cancelable` | [`Boolean!`](#boolean) | Specifies if a pipeline can be canceled. | | <a id="pipelinecodequalityreports"></a>`codeQualityReports` | [`CodeQualityDegradationConnection`](#codequalitydegradationconnection) | Code Quality degradations reported on the pipeline. (see [Connections](#connections)) | +| <a id="pipelinecommit"></a>`commit` | [`Commit`](#commit) | Git commit of the pipeline. | | <a id="pipelinecommitpath"></a>`commitPath` | [`String`](#string) | Path to the commit that triggered the pipeline. | | <a id="pipelinecommittedat"></a>`committedAt` | [`Time`](#time) | Timestamp of the pipeline's commit. | | <a id="pipelinecomplete"></a>`complete` | [`Boolean!`](#boolean) | Indicates if a pipeline is complete. | @@ -12388,6 +12641,7 @@ Represents a file or directory in the project repository that has been locked. | <a id="pipelinefinishedat"></a>`finishedAt` | [`Time`](#time) | Timestamp of the pipeline's completion. | | <a id="pipelineid"></a>`id` | [`ID!`](#id) | ID of the pipeline. | | <a id="pipelineiid"></a>`iid` | [`String!`](#string) | Internal ID of the pipeline. | +| <a id="pipelinejobartifacts"></a>`jobArtifacts` | [`[CiJobArtifact!]`](#cijobartifact) | Job artifacts of the pipeline. | | <a id="pipelinepath"></a>`path` | [`String`](#string) | Relative path to the pipeline's page. | | <a id="pipelineproject"></a>`project` | [`Project`](#project) | Project the pipeline belongs to. | | <a id="pipelinequeuedduration"></a>`queuedDuration` | [`Duration`](#duration) | How long the pipeline was queued before starting. | @@ -12559,8 +12813,8 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectcontainerexpirationpolicy"></a>`containerExpirationPolicy` | [`ContainerExpirationPolicy`](#containerexpirationpolicy) | Container expiration policy of the project. | | <a id="projectcontainerregistryenabled"></a>`containerRegistryEnabled` | [`Boolean`](#boolean) | Indicates if Container Registry is enabled for the current user. | | <a id="projectcontainerrepositoriescount"></a>`containerRepositoriesCount` | [`Int!`](#int) | Number of container repositories in the project. | +| <a id="projectcorpuses"></a>`corpuses` | [`CoverageFuzzingCorpusConnection`](#coveragefuzzingcorpusconnection) | Find corpuses of the project. Available only when feature flag `corpus_management` is enabled. This flag is disabled by default, because the feature is experimental and is subject to change without notice. (see [Connections](#connections)) | | <a id="projectcreatedat"></a>`createdAt` | [`Time`](#time) | Timestamp of the project creation. | -| <a id="projectdastprofiles"></a>`dastProfiles` | [`DastProfileConnection`](#dastprofileconnection) | DAST Profiles associated with the project. (see [Connections](#connections)) | | <a id="projectdastscannerprofiles"></a>`dastScannerProfiles` | [`DastScannerProfileConnection`](#dastscannerprofileconnection) | DAST scanner profiles associated with the project. (see [Connections](#connections)) | | <a id="projectdastsiteprofiles"></a>`dastSiteProfiles` | [`DastSiteProfileConnection`](#dastsiteprofileconnection) | DAST Site Profiles associated with the project. (see [Connections](#connections)) | | <a id="projectdescription"></a>`description` | [`String`](#string) | Short description of the project. | @@ -12580,6 +12834,7 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectjobsenabled"></a>`jobsEnabled` | [`Boolean`](#boolean) | Indicates if CI/CD pipeline jobs are enabled for the current user. | | <a id="projectlastactivityat"></a>`lastActivityAt` | [`Time`](#time) | Timestamp of the project last activity. | | <a id="projectlfsenabled"></a>`lfsEnabled` | [`Boolean`](#boolean) | Indicates if the project has Large File Storage (LFS) enabled. | +| <a id="projectmergecommittemplate"></a>`mergeCommitTemplate` | [`String`](#string) | Template used to create merge commit message in merge requests. | | <a id="projectmergerequestsenabled"></a>`mergeRequestsEnabled` | [`Boolean`](#boolean) | Indicates if Merge Requests are enabled for the current user. | | <a id="projectmergerequestsffonlyenabled"></a>`mergeRequestsFfOnlyEnabled` | [`Boolean`](#boolean) | Indicates if no merge commits should be created and all merges should instead be fast-forwarded, which means that merging is only allowed if the branch could be fast-forwarded. | | <a id="projectname"></a>`name` | [`String!`](#string) | Name of the project (without namespace). | @@ -12600,7 +12855,6 @@ Represents vulnerability finding of a security report on the pipeline. | <a id="projectrequestaccessenabled"></a>`requestAccessEnabled` | [`Boolean`](#boolean) | Indicates if users can request member access to the project. | | <a id="projectrequirementstatescount"></a>`requirementStatesCount` | [`RequirementStatesCount`](#requirementstatescount) | Number of requirements for the project by their state. | | <a id="projectsastciconfiguration"></a>`sastCiConfiguration` | [`SastCiConfiguration`](#sastciconfiguration) | SAST CI configuration for the project. | -| <a id="projectscanexecutionpolicies"></a>`scanExecutionPolicies` | [`ScanExecutionPolicyConnection`](#scanexecutionpolicyconnection) | Scan Execution Policies of the project. (see [Connections](#connections)) | | <a id="projectsecuritydashboardpath"></a>`securityDashboardPath` | [`String`](#string) | Path to project's security dashboard. | | <a id="projectsecurityscanners"></a>`securityScanners` | [`SecurityScanners`](#securityscanners) | Information about security analyzers used in the project. | | <a id="projectsentryerrors"></a>`sentryErrors` | [`SentryErrorCollection`](#sentryerrorcollection) | Paginated collection of Sentry errors on the project. | @@ -12798,8 +13052,25 @@ Returns [`DastProfile`](#dastprofile). | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="projectdastprofilehasdastprofileschedule"></a>`hasDastProfileSchedule` | [`Boolean`](#boolean) | Filter DAST Profiles by whether or not they have a schedule. Will be ignored if `dast_view_scans` feature flag is disabled. | | <a id="projectdastprofileid"></a>`id` | [`DastProfileID!`](#dastprofileid) | ID of the DAST Profile. | +##### `Project.dastProfiles` + +DAST Profiles associated with the project. + +Returns [`DastProfileConnection`](#dastprofileconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectdastprofileshasdastprofileschedule"></a>`hasDastProfileSchedule` | [`Boolean`](#boolean) | Filter DAST Profiles by whether or not they have a schedule. Will be ignored if `dast_view_scans` feature flag is disabled. | + ##### `Project.dastSiteProfile` DAST Site Profile associated with the project. @@ -12919,6 +13190,8 @@ Returns [`Issue`](#issue). | <a id="projectissuemilestonewildcardid"></a>`milestoneWildcardId` | [`MilestoneWildcardId`](#milestonewildcardid) | Filter issues by milestone ID wildcard. | | <a id="projectissuemyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. Wildcard values "NONE" and "ANY" are supported. | | <a id="projectissuenot"></a>`not` | [`NegatedIssueFilterInput`](#negatedissuefilterinput) | Negated arguments. | +| <a id="projectissuereleasetag"></a>`releaseTag` | [`[String!]`](#string) | Release tag associated with the issue's milestone. | +| <a id="projectissuereleasetagwildcardid"></a>`releaseTagWildcardId` | [`ReleaseTagWildcardId`](#releasetagwildcardid) | Filter issues by release tag ID wildcard. | | <a id="projectissuesearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="projectissuesort"></a>`sort` | [`IssueSort`](#issuesort) | Sort issues by this criteria. | | <a id="projectissuestate"></a>`state` | [`IssuableState`](#issuablestate) | Current state of this issue. | @@ -12953,6 +13226,8 @@ Returns [`IssueStatusCountsType`](#issuestatuscountstype). | <a id="projectissuestatuscountsmilestonewildcardid"></a>`milestoneWildcardId` | [`MilestoneWildcardId`](#milestonewildcardid) | Filter issues by milestone ID wildcard. | | <a id="projectissuestatuscountsmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. Wildcard values "NONE" and "ANY" are supported. | | <a id="projectissuestatuscountsnot"></a>`not` | [`NegatedIssueFilterInput`](#negatedissuefilterinput) | Negated arguments. | +| <a id="projectissuestatuscountsreleasetag"></a>`releaseTag` | [`[String!]`](#string) | Release tag associated with the issue's milestone. | +| <a id="projectissuestatuscountsreleasetagwildcardid"></a>`releaseTagWildcardId` | [`ReleaseTagWildcardId`](#releasetagwildcardid) | Filter issues by release tag ID wildcard. | | <a id="projectissuestatuscountssearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="projectissuestatuscountstypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filter issues by the given issue types. | | <a id="projectissuestatuscountsupdatedafter"></a>`updatedAfter` | [`Time`](#time) | Issues updated after this date. | @@ -12992,6 +13267,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectissuesmilestonewildcardid"></a>`milestoneWildcardId` | [`MilestoneWildcardId`](#milestonewildcardid) | Filter issues by milestone ID wildcard. | | <a id="projectissuesmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. Wildcard values "NONE" and "ANY" are supported. | | <a id="projectissuesnot"></a>`not` | [`NegatedIssueFilterInput`](#negatedissuefilterinput) | Negated arguments. | +| <a id="projectissuesreleasetag"></a>`releaseTag` | [`[String!]`](#string) | Release tag associated with the issue's milestone. | +| <a id="projectissuesreleasetagwildcardid"></a>`releaseTagWildcardId` | [`ReleaseTagWildcardId`](#releasetagwildcardid) | Filter issues by release tag ID wildcard. | | <a id="projectissuessearch"></a>`search` | [`String`](#string) | Search query for title or description. | | <a id="projectissuessort"></a>`sort` | [`IssueSort`](#issuesort) | Sort issues by this criteria. | | <a id="projectissuesstate"></a>`state` | [`IssuableState`](#issuablestate) | Current state of this issue. | @@ -13118,6 +13395,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="projectmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | | <a id="projectmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="projectmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="projectmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="projectmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="projectmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="projectmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -13219,6 +13498,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="projectpipelinesref"></a>`ref` | [`String`](#string) | Filter pipelines by the ref they are run for. | +| <a id="projectpipelinesscope"></a>`scope` | [`PipelineScopeEnum`](#pipelinescopeenum) | Filter pipelines by scope. | | <a id="projectpipelinessha"></a>`sha` | [`String`](#string) | Filter pipelines by the sha of the commit they are run for. | | <a id="projectpipelinessource"></a>`source` | [`String`](#string) | Filter pipelines by their source. Will be ignored if `dast_view_scans` feature flag is disabled. | | <a id="projectpipelinesstatus"></a>`status` | [`PipelineStatusEnum`](#pipelinestatusenum) | Filter pipelines by their status. | @@ -13308,6 +13588,22 @@ four standard [pagination arguments](#connection-pagination-arguments): | <a id="projectrequirementssort"></a>`sort` | [`Sort`](#sort) | List requirements by sort order. | | <a id="projectrequirementsstate"></a>`state` | [`RequirementState`](#requirementstate) | Filter requirements by state. | +##### `Project.scanExecutionPolicies` + +Scan Execution Policies of the project. + +Returns [`ScanExecutionPolicyConnection`](#scanexecutionpolicyconnection). + +This field returns a [connection](#connections). It accepts the +four standard [pagination arguments](#connection-pagination-arguments): +`before: String`, `after: String`, `first: Int`, `last: Int`. + +###### Arguments + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="projectscanexecutionpoliciesactionscantypes"></a>`actionScanTypes` | [`[SecurityReportTypeEnum!]`](#securityreporttypeenum) | Filters policies by the action scan type. Only these scan types are supported: `dast`, `secret_detection`, `cluster_image_scanning`, `container_scanning`. | + ##### `Project.sentryDetailedError` Detailed version of a Sentry error on the project. @@ -13402,6 +13698,7 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | +| <a id="projectvulnerabilitiesclusterid"></a>`clusterId` | [`[ClustersClusterID!]`](#clustersclusterid) | Filter vulnerabilities by `cluster_id`. Vulnerabilities with a `reportType` of `cluster_image_scanning` are only included with this filter. | | <a id="projectvulnerabilitieshasissues"></a>`hasIssues` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have linked issues. | | <a id="projectvulnerabilitieshasresolution"></a>`hasResolution` | [`Boolean`](#boolean) | Returns only the vulnerabilities which have been resolved on default branch. | | <a id="projectvulnerabilitiesimage"></a>`image` | [`[String!]`](#string) | Filter vulnerabilities by location image. When this filter is present, the response only matches entries for a `reportType` that includes `container_scanning`, `cluster_image_scanning`. | @@ -13733,7 +14030,7 @@ Returns [`[String!]`](#string). ##### `Repository.paginatedTree` -Paginated tree of the repository. Available only when feature flag `paginated_tree_graphql_query` is enabled. This flag is enabled by default. +Paginated tree of the repository. Returns [`TreeConnection`](#treeconnection). @@ -13782,6 +14079,7 @@ Returns [`Tree`](#tree). | <a id="repositoryblobname"></a>`name` | [`String`](#string) | Blob name. | | <a id="repositorybloboid"></a>`oid` | [`String!`](#string) | OID of the blob. | | <a id="repositoryblobpath"></a>`path` | [`String!`](#string) | Path of the blob. | +| <a id="repositoryblobpipelineeditorpath"></a>`pipelineEditorPath` | [`String`](#string) | Web path to edit .gitlab-ci.yml file. | | <a id="repositoryblobplaindata"></a>`plainData` | [`String`](#string) | Blob plain highlighted data. | | <a id="repositoryblobrawblob"></a>`rawBlob` | [`String`](#string) | Raw content of the blob. | | <a id="repositoryblobrawpath"></a>`rawPath` | [`String`](#string) | Web path to download the raw blob. | @@ -13912,7 +14210,7 @@ Counts of requirements by their state. | Name | Type | Description | | ---- | ---- | ----------- | | <a id="runnersetupinstallinstructions"></a>`installInstructions` | [`String!`](#string) | Instructions for installing the runner on the specified architecture. | -| <a id="runnersetupregisterinstructions"></a>`registerInstructions` | [`String`](#string) | Instructions for registering the runner. | +| <a id="runnersetupregisterinstructions"></a>`registerInstructions` | [`String`](#string) | Instructions for registering the runner. The actual registration tokens are not included in the commands. Instead, a placeholder `$REGISTRATION_TOKEN` is shown. | ### `SastCiConfiguration` @@ -14578,6 +14876,18 @@ Representing a to-do entry. | <a id="todostate"></a>`state` | [`TodoStateEnum!`](#todostateenum) | State of the to-do item. | | <a id="todotargettype"></a>`targetType` | [`TodoTargetEnum!`](#todotargetenum) | Target type of the to-do item. | +### `Topic` + +#### Fields + +| Name | Type | Description | +| ---- | ---- | ----------- | +| <a id="topicavatarurl"></a>`avatarUrl` | [`String`](#string) | URL to avatar image file of the topic. | +| <a id="topicdescription"></a>`description` | [`String`](#string) | Description of the topic. | +| <a id="topicdescriptionhtml"></a>`descriptionHtml` | [`String`](#string) | The GitLab Flavored Markdown rendering of `description`. | +| <a id="topicid"></a>`id` | [`ID!`](#id) | ID of the topic. | +| <a id="topicname"></a>`name` | [`String!`](#string) | Name of the topic. | + ### `Tree` #### Fields @@ -14688,6 +14998,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="usercoreassignedmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="usercoreassignedmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="usercoreassignedmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="usercoreassignedmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="usercoreassignedmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="usercoreassignedmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -14717,6 +15029,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="usercoreauthoredmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | +| <a id="usercoreauthoredmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="usercoreauthoredmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="usercoreauthoredmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="usercoreauthoredmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="usercoreauthoredmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -14764,6 +15078,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="usercorereviewrequestedmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | | <a id="usercorereviewrequestedmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="usercorereviewrequestedmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="usercorereviewrequestedmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="usercorereviewrequestedmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="usercorereviewrequestedmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="usercorereviewrequestedmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -16158,6 +16474,7 @@ State of a review of a GitLab merge request. | Value | Description | | ----- | ----------- | +| <a id="mergerequestreviewstateattention_requested"></a>`ATTENTION_REQUESTED` | The merge request is attention_requested. | | <a id="mergerequestreviewstatereviewed"></a>`REVIEWED` | The merge request is reviewed. | | <a id="mergerequestreviewstateunreviewed"></a>`UNREVIEWED` | The merge request is unreviewed. | @@ -16402,6 +16719,16 @@ Values for sorting package. | <a id="pipelineconfigsourceenumunknown_source"></a>`UNKNOWN_SOURCE` | Unknown source. | | <a id="pipelineconfigsourceenumwebide_source"></a>`WEBIDE_SOURCE` | Webide source. | +### `PipelineScopeEnum` + +| Value | Description | +| ----- | ----------- | +| <a id="pipelinescopeenumbranches"></a>`BRANCHES` | Branches. | +| <a id="pipelinescopeenumfinished"></a>`FINISHED` | Pipeline has completed. | +| <a id="pipelinescopeenumpending"></a>`PENDING` | Pipeline has not started running yet. | +| <a id="pipelinescopeenumrunning"></a>`RUNNING` | Pipeline is running. | +| <a id="pipelinescopeenumtags"></a>`TAGS` | Tags. | + ### `PipelineStatusEnum` | Value | Description | @@ -16462,6 +16789,15 @@ Values for sorting releases. | <a id="releasesortreleased_at_asc"></a>`RELEASED_AT_ASC` | Released at by ascending order. | | <a id="releasesortreleased_at_desc"></a>`RELEASED_AT_DESC` | Released at by descending order. | +### `ReleaseTagWildcardId` + +Release tag ID wildcard values. + +| Value | Description | +| ----- | ----------- | +| <a id="releasetagwildcardidany"></a>`ANY` | Release tag is assigned. | +| <a id="releasetagwildcardidnone"></a>`NONE` | No release tag is assigned. | + ### `RequirementState` State of a requirement. @@ -16511,6 +16847,7 @@ Size of UI component in SAST configuration page. | <a id="securityreporttypeenumdast"></a>`DAST` | DAST scan report. | | <a id="securityreporttypeenumdependency_scanning"></a>`DEPENDENCY_SCANNING` | DEPENDENCY SCANNING scan report. | | <a id="securityreporttypeenumsast"></a>`SAST` | SAST scan report. | +| <a id="securityreporttypeenumsast_iac"></a>`SAST_IAC` | SAST IAC scan report. | | <a id="securityreporttypeenumsecret_detection"></a>`SECRET_DETECTION` | SECRET DETECTION scan report. | ### `SecurityScannerType` @@ -16526,6 +16863,7 @@ The type of the security scanner. | <a id="securityscannertypedast"></a>`DAST` | DAST scanner. | | <a id="securityscannertypedependency_scanning"></a>`DEPENDENCY_SCANNING` | Dependency Scanning scanner. | | <a id="securityscannertypesast"></a>`SAST` | SAST scanner. | +| <a id="securityscannertypesast_iac"></a>`SAST_IAC` | Sast Iac scanner. | | <a id="securityscannertypesecret_detection"></a>`SECRET_DETECTION` | Secret Detection scanner. | ### `SentryErrorStatus` @@ -16572,12 +16910,14 @@ State of a Sentry error. | <a id="servicetypeprometheus_service"></a>`PROMETHEUS_SERVICE` | PrometheusService type. | | <a id="servicetypepushover_service"></a>`PUSHOVER_SERVICE` | PushoverService type. | | <a id="servicetyperedmine_service"></a>`REDMINE_SERVICE` | RedmineService type. | +| <a id="servicetypeshimo_service"></a>`SHIMO_SERVICE` | ShimoService type. | | <a id="servicetypeslack_service"></a>`SLACK_SERVICE` | SlackService type. | | <a id="servicetypeslack_slash_commands_service"></a>`SLACK_SLASH_COMMANDS_SERVICE` | SlackSlashCommandsService type. | | <a id="servicetypeteamcity_service"></a>`TEAMCITY_SERVICE` | TeamcityService type. | | <a id="servicetypeunify_circuit_service"></a>`UNIFY_CIRCUIT_SERVICE` | UnifyCircuitService type. | | <a id="servicetypewebex_teams_service"></a>`WEBEX_TEAMS_SERVICE` | WebexTeamsService type. | | <a id="servicetypeyoutrack_service"></a>`YOUTRACK_SERVICE` | YoutrackService type. | +| <a id="servicetypezentao_service"></a>`ZENTAO_SERVICE` | ZentaoService type. | ### `SharedRunnersSetting` @@ -16889,6 +17229,12 @@ A `AnalyticsDevopsAdoptionEnabledNamespaceID` is a global ID. It is encoded as a An example `AnalyticsDevopsAdoptionEnabledNamespaceID` is: `"gid://gitlab/Analytics::DevopsAdoption::EnabledNamespace/1"`. +### `AppSecFuzzingCoverageCorpusID` + +A `AppSecFuzzingCoverageCorpusID` is a global ID. It is encoded as a string. + +An example `AppSecFuzzingCoverageCorpusID` is: `"gid://gitlab/AppSec::Fuzzing::Coverage::Corpus/1"`. + ### `AuditEventsExternalAuditEventDestinationID` A `AuditEventsExternalAuditEventDestinationID` is a global ID. It is encoded as a string. @@ -17033,6 +17379,12 @@ An example `DastSiteValidationID` is: `"gid://gitlab/DastSiteValidation/1"`. Date represented in ISO 8601. +### `DependencyProxyManifestID` + +A `DependencyProxyManifestID` is a global ID. It is encoded as a string. + +An example `DependencyProxyManifestID` is: `"gid://gitlab/DependencyProxy::Manifest/1"`. + ### `DesignManagementDesignAtVersionID` A `DesignManagementDesignAtVersionID` is a global ID. It is encoded as a string. @@ -17632,6 +17984,7 @@ Represents metadata associated with a Package file. Implementations: - [`ConanFileMetadata`](#conanfilemetadata) +- [`HelmFileMetadata`](#helmfilemetadata) ##### Fields @@ -17733,6 +18086,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="userassignedmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="userassignedmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="userassignedmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="userassignedmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="userassignedmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="userassignedmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -17762,6 +18117,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | Name | Type | Description | | ---- | ---- | ----------- | | <a id="userauthoredmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | +| <a id="userauthoredmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="userauthoredmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="userauthoredmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="userauthoredmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="userauthoredmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -17809,6 +18166,8 @@ four standard [pagination arguments](#connection-pagination-arguments): | ---- | ---- | ----------- | | <a id="userreviewrequestedmergerequestsassigneeusername"></a>`assigneeUsername` | [`String`](#string) | Username of the assignee. | | <a id="userreviewrequestedmergerequestsauthorusername"></a>`authorUsername` | [`String`](#string) | Username of the author. | +| <a id="userreviewrequestedmergerequestscreatedafter"></a>`createdAfter` | [`Time`](#time) | Merge requests created after this timestamp. | +| <a id="userreviewrequestedmergerequestscreatedbefore"></a>`createdBefore` | [`Time`](#time) | Merge requests created before this timestamp. | | <a id="userreviewrequestedmergerequestsiids"></a>`iids` | [`[String!]`](#string) | Array of IIDs of merge requests, for example `[1, 2]`. | | <a id="userreviewrequestedmergerequestslabels"></a>`labels` | [`[String!]`](#string) | Array of label names. All resolved merge requests will have all of these labels. | | <a id="userreviewrequestedmergerequestsmergedafter"></a>`mergedAfter` | [`Time`](#time) | Merge requests merged after this date. | @@ -18161,6 +18520,7 @@ Represents an escalation rule. | <a id="negatedissuefilterinputmilestonetitle"></a>`milestoneTitle` | [`[String!]`](#string) | Milestone not applied to this issue. | | <a id="negatedissuefilterinputmilestonewildcardid"></a>`milestoneWildcardId` | [`NegatedMilestoneWildcardId`](#negatedmilestonewildcardid) | Filter by negated milestone wildcard values. | | <a id="negatedissuefilterinputmyreactionemoji"></a>`myReactionEmoji` | [`String`](#string) | Filter by reaction emoji applied by the current user. | +| <a id="negatedissuefilterinputreleasetag"></a>`releaseTag` | [`[String!]`](#string) | Release tag not associated with the issue's milestone. Ignored when parent is a group. | | <a id="negatedissuefilterinputtypes"></a>`types` | [`[IssueType!]`](#issuetype) | Filters out issues by the given issue types. | | <a id="negatedissuefilterinputweight"></a>`weight` | [`String`](#string) | Weight not applied to the issue. | diff --git a/doc/api/group_clusters.md b/doc/api/group_clusters.md index 81ca45c4531..eaecc74a96e 100644 --- a/doc/api/group_clusters.md +++ b/doc/api/group_clusters.md @@ -4,9 +4,13 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Group clusters API **(FREE)** +# Group clusters API (certificate-based) (DEPRECATED) **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30213) in GitLab 12.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30213) in GitLab 12.1. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. Similarly to [project-level](../user/project/clusters/index.md) and [instance-level](../user/instance/clusters/index.md) Kubernetes clusters, diff --git a/doc/api/group_repository_storage_moves.md b/doc/api/group_repository_storage_moves.md index 9d4120ec355..1d6e08b5840 100644 --- a/doc/api/group_repository_storage_moves.md +++ b/doc/api/group_repository_storage_moves.md @@ -11,7 +11,7 @@ type: reference Group repositories can be moved between storages. This API can help you when [migrating to Gitaly Cluster](../administration/gitaly/index.md#migrating-to-gitaly-cluster), for -example, or to migrate a [group wiki](../user/project/wiki/index.md#group-wikis). +example, or to migrate a [group wiki](../user/project/wiki/group.md). As group repository storage moves are processed, they transition through different states. Values of `state` are: diff --git a/doc/api/group_wikis.md b/doc/api/group_wikis.md index 17337934a92..4af907bd387 100644 --- a/doc/api/group_wikis.md +++ b/doc/api/group_wikis.md @@ -9,7 +9,7 @@ type: reference, api > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/212199) in GitLab 13.5. -The [group wikis](../user/project/wiki/index.md#group-wikis) API is available only in APIv4. +The [group wikis](../user/project/wiki/group.md) API is available only in APIv4. An API for [project wikis](wikis.md) is also available. ## List wiki pages diff --git a/doc/api/groups.md b/doc/api/groups.md index 7efecfc2c9c..5faa63585c1 100644 --- a/doc/api/groups.md +++ b/doc/api/groups.md @@ -100,13 +100,15 @@ GET /groups?statistics=true "parent_id": null, "created_at": "2020-01-15T12:36:29.590Z", "statistics": { - "storage_size" : 363, - "repository_size" : 33, - "wiki_size" : 100, - "lfs_objects_size" : 123, - "job_artifacts_size" : 57, + "storage_size": 363, + "repository_size": 33, + "wiki_size": 100, + "lfs_objects_size": 123, + "job_artifacts_size": 57, + "pipeline_artifacts_size": 0, "packages_size": 0, - "snippets_size" : 50 + "snippets_size": 50, + "uploads_size": 0 } } ] diff --git a/doc/api/import.md b/doc/api/import.md index d556c8af971..18c0eb04fff 100644 --- a/doc/api/import.md +++ b/doc/api/import.md @@ -83,3 +83,8 @@ curl --request POST \ "bitbucket_server_repo": "my-repo" }' ``` + +## Automate group and project import **(PREMIUM)** + +For information on automating user, group, and project import API calls, see +[Automate group and project import](../user/project/import/index.md#automate-group-and-project-import). diff --git a/doc/api/instance_clusters.md b/doc/api/instance_clusters.md index 58f88b26bc4..85046388275 100644 --- a/doc/api/instance_clusters.md +++ b/doc/api/instance_clusters.md @@ -4,9 +4,13 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Instance clusters API **(FREE SELF)** +# Instance clusters API (certificate-based) (DEPRECATED) **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36001) in GitLab 13.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36001) in GitLab 13.2. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. With [instance-level Kubernetes clusters](../user/instance/clusters/index.md), you can connect a Kubernetes cluster to the GitLab instance and use the same cluster across all of diff --git a/doc/api/integrations.md b/doc/api/integrations.md index 3c649e8d044..8f57e915b4e 100644 --- a/doc/api/integrations.md +++ b/doc/api/integrations.md @@ -94,9 +94,9 @@ Parameters: | `restrict_to_branch` | string | false | Comma-separated list of branches to be are automatically inspected. Leave blank to include all branches. | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Asana integration +### Disable Asana integration -Delete Asana integration for a project. +Disable the Asana integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/asana @@ -130,9 +130,9 @@ Parameters: | `subdomain` | string | false | The subdomain setting | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Assembla integration +### Disable Assembla integration -Delete Assembla integration for a project. +Disable the Assembla integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/assembla @@ -170,9 +170,9 @@ Parameters: | `password` | string | true | Password of the user | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Atlassian Bamboo CI integration +### Disable Atlassian Bamboo CI integration -Delete Atlassian Bamboo CI integration for a project. +Disable the Atlassian Bamboo CI integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/bamboo @@ -209,9 +209,9 @@ Parameters: | `title` | string | false | Title | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Bugzilla integration +### Disable Bugzilla integration -Delete Bugzilla integration for a project. +Disable the Bugzilla integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/bugzilla @@ -246,9 +246,9 @@ Parameters: | `enable_ssl_verification` | boolean | false | DEPRECATED: This parameter has no effect since SSL verification is always enabled | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Buildkite integration +### Disable Buildkite integration -Delete Buildkite integration for a project. +Disable the Buildkite integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/buildkite @@ -284,9 +284,9 @@ Parameters: | `room` | string | false | Campfire room. The last part of the URL when you're in a room. | | `push_events` | boolean | false | Enable notifications for push events. | -### Delete Campfire integration +### Disable Campfire integration -Delete Campfire integration for a project. +Disable the Campfire integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/campfire @@ -322,9 +322,9 @@ Parameters: | `datadog_service` | string | false | Name of this GitLab instance that all data will be tagged with | | `datadog_env` | string | false | The environment tag that traces will be tagged with | -### Delete Datadog integration +### Disable Datadog integration -Delete Datadog integration for a project. +Disable the Datadog integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/datadog @@ -367,9 +367,9 @@ Parameters: | `pipeline_events` | boolean | false | Enable notifications for pipeline events | | `wiki_page_events` | boolean | false | Enable notifications for wiki page events | -### Delete Unify Circuit integration +### Disable Unify Circuit integration -Delete Unify Circuit integration for a project. +Disable the Unify Circuit integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/unify-circuit @@ -412,9 +412,9 @@ Parameters: | `pipeline_events` | boolean | false | Enable notifications for pipeline events | | `wiki_page_events` | boolean | false | Enable notifications for wiki page events | -### Delete Webex Teams integration +### Disable Webex Teams integration -Delete Webex Teams integration for a project. +Disable the Webex Teams integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/webex-teams @@ -451,9 +451,9 @@ Parameters: | `title` | string | false | Title | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Custom Issue Tracker integration +### Disable Custom Issue Tracker integration -Delete Custom Issue Tracker integration for a project. +Disable the Custom Issue Tracker integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/custom-issue-tracker @@ -485,9 +485,9 @@ Parameters: | --------- | ---- | -------- | ----------- | | `webhook` | string | true | Discord webhook. For example, `https://discord.com/api/webhooks/…` | -### Delete Discord integration +### Disable Discord integration -Delete Discord integration for a project. +Disable the Discord integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/discord @@ -524,9 +524,9 @@ Parameters: | `merge_requests_events` | boolean | false | Enable notifications for merge request events | | `tag_push_events` | boolean | false | Enable notifications for tag push events | -### Delete Drone CI integration +### Disable Drone CI integration -Delete Drone CI integration for a project. +Disable the Drone CI integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/drone-ci @@ -563,9 +563,9 @@ Parameters: | `tag_push_events` | boolean | false | Enable notifications for tag push events | | `branches_to_be_notified` | string | false | Branches to send notifications for. Valid options are "all", "default", "protected", and "default_and_protected". Notifications are always fired for tag pushes. The default value is "all" | -### Delete Emails on Push integration +### Disable Emails on Push integration -Delete Emails on Push integration for a project. +Disable the Emails on Push integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/emails-on-push @@ -599,9 +599,9 @@ Parameters: | `project_url` | string | true | The URL to the project in EWM | | `issues_url` | string | true | The URL to view an issue in EWM. Must contain `:id` | -### Delete EWM integration +### Disable EWM integration -Delete EWM integration for a project. +Disable the EWM integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/ewm @@ -635,9 +635,9 @@ Parameters: | --------- | ---- | -------- | ----------- | | `confluence_url` | string | true | The URL of the Confluence Cloud Workspace hosted on atlassian.net. | -### Delete Confluence integration +### Disable Confluence integration -Delete Confluence integration for a project. +Disable the Confluence integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/confluence @@ -669,9 +669,9 @@ Parameters: | --------- | ---- | -------- | ----------- | | `external_wiki_url` | string | true | The URL of the external wiki | -### Delete External wiki integration +### Disable External wiki integration -Delete External wiki integration for a project. +Disable the External wiki integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/external-wiki @@ -706,9 +706,9 @@ Parameters: | `token` | string | true | Flowdock Git source token | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Flowdock integration +### Disable Flowdock integration -Delete Flowdock integration for a project. +Disable the Flowdock integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/flowdock @@ -740,11 +740,11 @@ Parameters: | --------- | ---- | -------- | ----------- | | `token` | string | true | GitHub API token with `repo:status` OAuth scope | | `repository_url` | string | true | GitHub repository URL | -| `static_context` | boolean | false | Append instance name instead of branch to [status check name](../user/project/integrations/github.md#static--dynamic-status-check-names) | +| `static_context` | boolean | false | Append instance name instead of branch to [status check name](../user/project/integrations/github.md#static-or-dynamic-status-check-names) | -### Delete GitHub integration +### Disable GitHub integration -Delete GitHub integration for a project. +Disable the GitHub integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/github @@ -788,9 +788,9 @@ Parameters: | `pipeline_events` | boolean | false | Enable notifications for pipeline events | | `wiki_page_events` | boolean | false | Enable notifications for wiki page events | -### Delete Hangouts Chat integration +### Disable Hangouts Chat integration -Delete Hangouts Chat integration for a project. +Disable the Hangouts Chat integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/hangouts-chat @@ -829,9 +829,9 @@ Parameters: | `colorize_messages` | boolean | false | Colorize messages | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Irker (IRC gateway) integration +### Disable Irker (IRC gateway) integration -Delete Irker (IRC gateway) integration for a project. +Disable the Irker (IRC gateway) integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/irker @@ -880,9 +880,9 @@ Parameters: | `merge_requests_events` | boolean | false | Enable notifications for merge request events | | `comment_on_event_enabled` | boolean | false | Enable comments inside Jira issues on each GitLab event (commit / merge request) | -### Delete Jira integration +### Disable Jira integration -Remove all previously Jira integrations from a project. +Disable the Jira integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/jira @@ -939,9 +939,9 @@ Parameters: | --------- | ---- | -------- | ----------- | | `token` | string | yes | The Slack token | -### Delete Slack Slash Command integration +### Disable Slack Slash Command integration -Delete Slack Slash Command integration for a project. +Disable the Slack Slash Command integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/slack-slash-commands @@ -974,9 +974,9 @@ Parameters: | `token` | string | yes | The Mattermost token | | `username` | string | no | The username to use to post the message | -### Delete Mattermost Slash Command integration +### Disable Mattermost Slash Command integration -Delete Mattermost Slash Command integration for a project. +Disable the Mattermost Slash Command integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/mattermost-slash-commands @@ -1005,9 +1005,9 @@ Parameters: | `merge_requests_events` | boolean | false | Enable notifications for merge request events | | `tag_push_events` | boolean | false | Enable notifications for tag push events | -### Delete Packagist integration +### Disable Packagist integration -Delete Packagist integration for a project. +Disable the Packagist integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/packagist @@ -1044,9 +1044,9 @@ Parameters: | `notify_only_default_branch` | boolean | no | Send notifications only for the default branch ([introduced in GitLab 12.0](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/28271)) | | `pipeline_events` | boolean | false | Enable notifications for pipeline events | -### Delete Pipeline-Emails integration +### Disable Pipeline-Emails integration -Delete Pipeline-Emails integration for a project. +Disable the Pipeline-Emails integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/pipelines-email @@ -1082,9 +1082,9 @@ Parameters: | `restrict_to_branch` | boolean | false | Comma-separated list of branches to automatically inspect. Leave blank to include all branches. | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Pivotal Tracker integration +### Disable Pivotal Tracker integration -Delete Pivotal Tracker integration for a project. +Disable the Pivotal Tracker integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/pivotaltracker @@ -1118,9 +1118,9 @@ Parameters: | `google_iap_audience_client_id` | string | false | Client ID of the IAP secured resource (looks like IAP_CLIENT_ID.apps.googleusercontent.com) | | `google_iap_service_account_json` | string | false | `credentials.json` file for your service account, like { "type": "service_account", "project_id": ... } | -### Delete Prometheus integration +### Disable Prometheus integration -Delete Prometheus integration for a project. +Disable the Prometheus integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/prometheus @@ -1157,9 +1157,9 @@ Parameters: | `sound` | string | false | The sound of the notification | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Pushover integration +### Disable Pushover integration -Delete Pushover integration for a project. +Disable the Pushover integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/pushover @@ -1195,9 +1195,9 @@ Parameters: | `description` | string | false | Description | | `push_events` | boolean | false | Enable notifications for push events | -### Delete Redmine integration +### Disable Redmine integration -Delete Redmine integration for a project. +Disable the Redmine integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/redmine @@ -1256,9 +1256,9 @@ Parameters: | `wiki_page_channel` | string | false | The name of the channel to receive wiki page events notifications | | `wiki_page_events` | boolean | false | Enable notifications for wiki page events | -### Delete Slack integration +### Disable Slack integration -Delete Slack integration for a project. +Disable the Slack integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/slack @@ -1302,9 +1302,9 @@ Parameters: | `pipeline_events` | boolean | false | Enable notifications for pipeline events | | `wiki_page_events` | boolean | false | Enable notifications for wiki page events | -### Delete Microsoft Teams integration +### Disable Microsoft Teams integration -Delete Microsoft Teams integration for a project. +Disable the Microsoft Teams integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/microsoft-teams @@ -1359,9 +1359,9 @@ Parameters: | `pipeline_channel` | string | false | The name of the channel to receive pipeline events notifications | | `wiki_page_channel` | string | false | The name of the channel to receive wiki page events notifications | -### Delete Mattermost notifications integration +### Disable Mattermost notifications integration -Delete Mattermost notifications integration for a project. +Disable the Mattermost notifications integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/mattermost @@ -1399,9 +1399,9 @@ Parameters: | `password` | string | true | The password of the user | | `push_events` | boolean | false | Enable notifications for push events | -### Delete JetBrains TeamCity CI integration +### Disable JetBrains TeamCity CI integration -Delete JetBrains TeamCity CI integration for a project. +Disable the JetBrains TeamCity CI integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/teamcity @@ -1439,9 +1439,9 @@ Parameters: | `merge_requests_events` | boolean | false | Enable notifications for merge request events. | | `tag_push_events` | boolean | false | Enable notifications for tag push events. | -### Delete Jenkins CI integration +### Disable Jenkins CI integration -Delete Jenkins CI integration for a project. +Disable the Jenkins CI integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/jenkins @@ -1476,9 +1476,9 @@ Parameters: - `multiproject_enabled` (optional) - Multi-project mode is configured in Jenkins GitLab Hook plugin - `pass_unstable` (optional) - Unstable builds are treated as passing -### Delete Jenkins CI (Deprecated) integration +### Disable Jenkins CI (Deprecated) integration -Delete Jenkins CI (Deprecated) integration for a project. +Disable the Jenkins CI (Deprecated) integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/jenkins-deprecated @@ -1512,9 +1512,9 @@ Parameters: | --------- | ---- | -------- | ----------- | | `mock_service_url` | string | true | `http://localhost:4004` | -### Delete MockCI integration +### Disable MockCI integration -Delete MockCI integration for a project. +Disable the MockCI integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/mock-ci @@ -1549,9 +1549,9 @@ Parameters: | `description` | string | false | Description | | `push_events` | boolean | false | Enable notifications for push events | -### Delete YouTrack integration +### Disable YouTrack integration -Delete YouTrack integration for a project. +Disable the YouTrack integration for a project. Integration settings are preserved. ```plaintext DELETE /projects/:id/integrations/youtrack diff --git a/doc/api/invitations.md b/doc/api/invitations.md index 26e85a9d9f3..80a05f8ea0d 100644 --- a/doc/api/invitations.md +++ b/doc/api/invitations.md @@ -43,6 +43,8 @@ POST /projects/:id/invitations | `expires_at` | string | no | A date string in the format YEAR-MONTH-DAY | | `invite_source` | string | no | The source of the invitation that starts the member creation process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/327120). | | `areas_of_focus` | string | no | Areas the inviter wants the member to focus upon. | +| `tasks_to_be_done` | array of strings | no | Tasks the inviter wants the member to focus on. The tasks are added as issues to a specified project. The possible values are: `ci`, `code` and `issues`. If specified, requires `tasks_project_id`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `invite_members_for_task`. Disabled by default. | +| `tasks_project_id` | integer | no | The project ID in which to create the task issues. If specified, requires `tasks_to_be_done`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `invite_members_for_task`. Disabled by default. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ diff --git a/doc/api/jobs.md b/doc/api/jobs.md index ac8b756beac..2a07e2d92c5 100644 --- a/doc/api/jobs.md +++ b/doc/api/jobs.md @@ -294,7 +294,7 @@ Example of response ] ``` -In GitLab 13.3 and later, this endpoint [returns data for any pipeline](pipelines.md#single-pipeline-requests) +In GitLab 13.3 and later, this endpoint [returns data for any pipeline](pipelines.md#get-a-single-pipeline) including [child pipelines](../ci/pipelines/parent_child_pipelines.md). In GitLab 13.5 and later, this endpoint does not return retried jobs in the response @@ -400,11 +400,12 @@ Retrieve the job that generated a job token. GET /job ``` -Examples +Examples (must run as part of the [`script`](../ci/yaml/index.md#script) section of a [CI/CD job](../ci/jobs/index.md)): ```shell -curl --header "JOB-TOKEN: <your_job_token>" "https://gitlab.example.com/api/v4/job" -curl "https://gitlab.example.com/api/v4/job?job_token=<your_job_token>" +curl --header "Authorization: Bearer $CI_JOB_TOKEN" "${CI_API_V4_URL}/job" +curl --header "JOB-TOKEN: $CI_JOB_TOKEN" "${CI_API_V4_URL}/job" +curl "${CI_API_V4_URL}/job?job_token=$CI_JOB_TOKEN" ``` Example of response diff --git a/doc/api/lint.md b/doc/api/lint.md index 9f95b9a94ae..e5b5e0e2be8 100644 --- a/doc/api/lint.md +++ b/doc/api/lint.md @@ -30,6 +30,7 @@ POST /ci/lint | ---------- | ------- | -------- | -------- | | `content` | string | yes | The CI/CD configuration content. | | `include_merged_yaml` | boolean | no | If the [expanded CI/CD configuration](#yaml-expansion) should be included in the response. | +| `include_jobs` | boolean | no | If the list of jobs should be included in the response. This is false by default. | ```shell curl --header "Content-Type: application/json" --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/ci/lint" --data '{"content": "{ \"image\": \"ruby:2.6\", \"services\": [\"postgres\"], \"before_script\": [\"bundle install\", \"bundle exec rake db:create\"], \"variables\": {\"DB_NAME\": \"postgres\"}, \"types\": [\"test\", \"deploy\", \"notify\"], \"rspec\": { \"script\": \"rake spec\", \"tags\": [\"ruby\", \"postgres\"], \"only\": [\"branches\"]}}"}' @@ -91,7 +92,7 @@ work for CI configuration added with [`include: local`](../ci/yaml/index.md#incl or with [`extends:`](../ci/yaml/index.md#extends). Example contents of a `.gitlab-ci.yml` passed to the CI Lint API with -`include_merged_yaml` set as true: +`include_merged_yaml` and `include_jobs` set as true: ```yaml include: @@ -118,7 +119,39 @@ Example response: { "status": "valid", "errors": [], - "merged_yaml": "---\n:another_test:\n :stage: test\n :script: echo 2\n:test:\n :stage: test\n :script: echo 1\n" + "merged_yaml": "---\n:another_test:\n :stage: test\n :script: echo 2\n:test:\n :stage: test\n :script: echo 1\n", + "jobs": [ + { + "name":"test", + "stage":"test", + "before_script":[], + "script":["echo 1"], + "after_script":[], + "tag_list":[], + "environment":null, + "when":"on_success", + "allow_failure":false, + "only":{ + "refs":["branches","tags"] + }, + "except":null + }, + { + "name":"another_test", + "stage":"test", + "before_script":[], + "script":["echo 2"], + "after_script":[], + "tag_list":[], + "environment":null, + "when":"on_success", + "allow_failure":false, + "only":{ + "refs":["branches","tags"] + }, + "except":null + } + ] } ``` @@ -137,6 +170,7 @@ POST /projects/:id/ci/lint | ---------- | ------- | -------- | -------- | | `content` | string | yes | The CI/CD configuration content. | | `dry_run` | boolean | no | Run [pipeline creation simulation](../ci/lint.md#pipeline-simulation), or only do static check. This is false by default. | +| `include_jobs` | boolean | no | If the list of jobs that would exist in a static check or pipeline simulation should be included in the response. This is false by default. | Example request: @@ -185,6 +219,7 @@ GET /projects/:id/ci/lint | Attribute | Type | Required | Description | | ---------- | ------- | -------- | -------- | | `dry_run` | boolean | no | Run pipeline creation simulation, or only do static check. | +| `include_jobs` | boolean | no | If the list of jobs that would exist in a static check or pipeline simulation should be included in the response. This is false by default. | Example request: diff --git a/doc/api/members.md b/doc/api/members.md index 44e58f49d3b..ce276487f21 100644 --- a/doc/api/members.md +++ b/doc/api/members.md @@ -422,6 +422,8 @@ POST /projects/:id/members | `expires_at` | string | no | A date string in the format `YEAR-MONTH-DAY` | | `invite_source` | string | no | The source of the invitation that starts the member creation process. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/327120). | | `areas_of_focus` | string | no | Areas the inviter wants the member to focus upon. | +| `tasks_to_be_done` | array of strings | no | Tasks the inviter wants the member to focus on. The tasks are added as issues to a specified project. The possible values are: `ci`, `code` and `issues`. If specified, requires `tasks_project_id`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `invite_members_for_task`. Disabled by default. | +| `tasks_project_id` | integer | no | The project ID in which to create the task issues. If specified, requires `tasks_to_be_done`. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69299) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `invite_members_for_task`. Disabled by default. | ```shell curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" \ @@ -587,6 +589,43 @@ curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://git curl --request DELETE --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/:id/members/:user_id" ``` +## Approve a member for a group + +Approves a pending user for a group and its subgroups and projects. + +```plaintext +PUT /groups/:id/members/:user_id/approve +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the root group](index.md#namespaced-path-encoding) owned by the authenticated user | +| `user_id` | integer | yes | The user ID of the member | + +Example request: + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/members/:user_id/approve" +``` + +## Approve all pending members for a group + +Approves all pending users for a group and its subgroups and projects. + +```plaintext +POST /groups/:id/members/approve_all +``` + +| Attribute | Type | Required | Description | +| --------- | ---- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the root group](index.md#namespaced-path-encoding) owned by the authenticated user | + +Example request: + +```shell +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/:id/members/approve_all" +``` + ## Give a group access to a project See [share project with group](projects.md#share-project-with-group) diff --git a/doc/api/packages/debian_group_distributions.md b/doc/api/packages/debian_group_distributions.md index c5d2effcf44..0a1b7b4571e 100644 --- a/doc/api/packages/debian_group_distributions.md +++ b/doc/api/packages/debian_group_distributions.md @@ -107,6 +107,43 @@ Example response: } ``` +## Single Debian group distribution key + +Gets a single Debian group distribution key. + +```plaintext +GET /groups/:id/debian_distributions/:codename/key.asc +``` + +| Attribute | Type | Required | Description | +| ---------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the group](../index.md#namespaced-path-encoding) owned by the authenticated user. | +| `codename` | integer | yes | The `codename` of a distribution. | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/groups/5/debian_distributions/unstable/key.asc" +``` + +Example response: + +```plaintext +-----BEGIN PGP PUBLIC KEY BLOCK----- +Comment: Alice's OpenPGP certificate +Comment: https://www.ietf.org/id/draft-bre-openpgp-samples-01.html + +mDMEXEcE6RYJKwYBBAHaRw8BAQdArjWwk3FAqyiFbFBKT4TzXcVBqPTB3gmzlC/U +b7O1u120JkFsaWNlIExvdmVsYWNlIDxhbGljZUBvcGVucGdwLmV4YW1wbGU+iJAE +ExYIADgCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQTrhbtfozp14V6UTmPy +MVUMT0fjjgUCXaWfOgAKCRDyMVUMT0fjjukrAPoDnHBSogOmsHOsd9qGsiZpgRnO +dypvbm+QtXZqth9rvwD9HcDC0tC+PHAsO7OTh1S1TC9RiJsvawAfCPaQZoed8gK4 +OARcRwTpEgorBgEEAZdVAQUBAQdAQv8GIa2rSTzgqbXCpDDYMiKRVitCsy203x3s +E9+eviIDAQgHiHgEGBYIACAWIQTrhbtfozp14V6UTmPyMVUMT0fjjgUCXEcE6QIb +DAAKCRDyMVUMT0fjjlnQAQDFHUs6TIcxrNTtEZFjUFm1M0PJ1Dng/cDW4xN80fsn +0QEA22Kr7VkCjeAEC08VSTeV+QFsmz55/lntWkwYWhmvOgE= +=iIGO +-----END PGP PUBLIC KEY BLOCK----- +``` + ## Create a Debian group distribution Creates a Debian group distribution. diff --git a/doc/api/packages/debian_project_distributions.md b/doc/api/packages/debian_project_distributions.md index bedf3f1f27a..533742642fd 100644 --- a/doc/api/packages/debian_project_distributions.md +++ b/doc/api/packages/debian_project_distributions.md @@ -106,6 +106,43 @@ Example response: } ``` +## Single Debian project distribution key + +Gets a single Debian project distribution key. + +```plaintext +GET /projects/:id/debian_distributions/:codename/key.asc +``` + +| Attribute | Type | Required | Description | +| ---------- | -------------- | -------- | ----------- | +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../index.md#namespaced-path-encoding) owned by the authenticated user. | +| `codename` | integer | yes | The `codename` of a distribution. | + +```shell +curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/5/debian_distributions/unstable/key.asc" +``` + +Example response: + +```plaintext +-----BEGIN PGP PUBLIC KEY BLOCK----- +Comment: Alice's OpenPGP certificate +Comment: https://www.ietf.org/id/draft-bre-openpgp-samples-01.html + +mDMEXEcE6RYJKwYBBAHaRw8BAQdArjWwk3FAqyiFbFBKT4TzXcVBqPTB3gmzlC/U +b7O1u120JkFsaWNlIExvdmVsYWNlIDxhbGljZUBvcGVucGdwLmV4YW1wbGU+iJAE +ExYIADgCGwMFCwkIBwIGFQoJCAsCBBYCAwECHgECF4AWIQTrhbtfozp14V6UTmPy +MVUMT0fjjgUCXaWfOgAKCRDyMVUMT0fjjukrAPoDnHBSogOmsHOsd9qGsiZpgRnO +dypvbm+QtXZqth9rvwD9HcDC0tC+PHAsO7OTh1S1TC9RiJsvawAfCPaQZoed8gK4 +OARcRwTpEgorBgEEAZdVAQUBAQdAQv8GIa2rSTzgqbXCpDDYMiKRVitCsy203x3s +E9+eviIDAQgHiHgEGBYIACAWIQTrhbtfozp14V6UTmPyMVUMT0fjjgUCXEcE6QIb +DAAKCRDyMVUMT0fjjlnQAQDFHUs6TIcxrNTtEZFjUFm1M0PJ1Dng/cDW4xN80fsn +0QEA22Kr7VkCjeAEC08VSTeV+QFsmz55/lntWkwYWhmvOgE= +=iIGO +-----END PGP PUBLIC KEY BLOCK----- +``` + ## Create a Debian project distribution Creates a Debian project distribution. diff --git a/doc/api/packages/maven.md b/doc/api/packages/maven.md index b4b3d579ffb..b046b0dc411 100644 --- a/doc/api/packages/maven.md +++ b/doc/api/packages/maven.md @@ -36,13 +36,13 @@ GET packages/maven/*path/:file_name | `file_name` | string | yes | The name of the Maven package file. | ```shell -curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/packages/maven/foo/bar/mypkg/1.0-SNAPSHOT/mypkg-1.0-SNAPSHOT.jar" ``` To write the output to file: ```shell -curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" >> mypkg-1.0-SNAPSHOT.jar +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/packages/maven/foo/bar/mypkg/1.0-SNAPSHOT/mypkg-1.0-SNAPSHOT.jar" >> mypkg-1.0-SNAPSHOT.jar ``` This writes the downloaded file to `mypkg-1.0-SNAPSHOT.jar` in the current directory. @@ -63,13 +63,13 @@ GET groups/:id/-/packages/maven/*path/:file_name | `file_name` | string | yes | The name of the Maven package file. | ```shell -curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/groups/1/-/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/groups/1/-/packages/maven/foo/bar/mypkg/1.0-SNAPSHOT/mypkg-1.0-SNAPSHOT.jar" ``` To write the output to file: ```shell -curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/groups/1/-/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" >> mypkg-1.0-SNAPSHOT.jar +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/groups/1/-/packages/maven/foo/bar/mypkg/1.0-SNAPSHOT/mypkg-1.0-SNAPSHOT.jar" >> mypkg-1.0-SNAPSHOT.jar ``` This writes the downloaded file to `mypkg-1.0-SNAPSHOT.jar` in the current directory. @@ -90,13 +90,13 @@ GET projects/:id/packages/maven/*path/:file_name | `file_name` | string | yes | The name of the Maven package file. | ```shell -curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/maven/foo/bar/mypkg/1.0-SNAPSHOT/mypkg-1.0-SNAPSHOT.jar" ``` To write the output to file: ```shell -curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.jar" >> mypkg-1.0-SNAPSHOT.jar +curl --header "Private-Token: <personal_access_token>" "https://gitlab.example.com/api/v4/projects/1/packages/maven/foo/bar/mypkg/1.0-SNAPSHOT/mypkg-1.0-SNAPSHOT.jar" >> mypkg-1.0-SNAPSHOT.jar ``` This writes the downloaded file to `mypkg-1.0-SNAPSHOT.jar` in the current directory. @@ -120,5 +120,5 @@ PUT projects/:id/packages/maven/*path/:file_name curl --request PUT \ --upload-file path/to/mypkg-1.0-SNAPSHOT.pom \ --header "Private-Token: <personal_access_token>" \ - "https://gitlab.example.com/api/v4/projects/1/packages/maven/foo/bar/baz/mypkg-1.0-SNAPSHOT.pom" + "https://gitlab.example.com/api/v4/projects/1/packages/maven/foo/bar/mypkg/1.0-SNAPSHOT/mypkg-1.0-SNAPSHOT.pom" ``` diff --git a/doc/api/pipelines.md b/doc/api/pipelines.md index f3c30a414ea..4dddbbc7826 100644 --- a/doc/api/pipelines.md +++ b/doc/api/pipelines.md @@ -6,14 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Pipelines API **(FREE)** -## Single Pipeline Requests - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36494) in GitLab 13.3. - -Endpoints that request information about a single pipeline return data for any pipeline. -Before 13.3, requests for [child pipelines](../ci/pipelines/parent_child_pipelines.md) returned -a 404 error. - ## Pipelines pagination By default, `GET` requests return 20 results at a time because the API results @@ -23,6 +15,9 @@ Read more on [pagination](index.md#pagination). ## List project pipelines +List pipelines in a project. Child pipelines are not included in the results, +but you can [get child pipeline](pipelines.md#get-a-single-pipeline) individually. + ```plaintext GET /projects/:id/pipelines ``` @@ -79,6 +74,11 @@ Example of response ## Get a single pipeline +Get one pipeline from a project. + +You can also get a single [child pipeline](../ci/pipelines/parent_child_pipelines.md). +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36494) in GitLab 13.3. + ```plaintext GET /projects/:id/pipelines/:pipeline_id ``` diff --git a/doc/api/project_clusters.md b/doc/api/project_clusters.md index bb05e4788d0..d2a574b5cbd 100644 --- a/doc/api/project_clusters.md +++ b/doc/api/project_clusters.md @@ -4,9 +4,13 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Project clusters API **(FREE)** +# Project clusters API (certificate-based) (DEPRECATED) **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23922) in GitLab 11.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23922) in GitLab 11.7. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. Users need at least the [Maintainer](../user/permissions.md) role to use these endpoints. diff --git a/doc/api/projects.md b/doc/api/projects.md index 024362f3246..d19019c9597 100644 --- a/doc/api/projects.md +++ b/doc/api/projects.md @@ -182,6 +182,7 @@ When the user is authenticated and `simple` is not set this returns something li "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "marked_for_deletion_at": "2020-04-03", // Deprecated and will be removed in API v5 in favor of marked_for_deletion_on "marked_for_deletion_on": "2020-04-03", "statistics": { @@ -191,8 +192,10 @@ When the user is authenticated and `simple` is not set this returns something li "wiki_size" : 0, "lfs_objects_size": 0, "job_artifacts_size": 0, + "pipeline_artifacts_size": 0, "packages_size": 0, - "snippets_size": 0 + "snippets_size": 0, + "uploads_size": 0 }, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-client", "_links": { @@ -296,6 +299,7 @@ When the user is authenticated and `simple` is not set this returns something li "service_desk_address": null, "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "statistics": { "commit_count": 12, "storage_size": 2066080, @@ -303,8 +307,10 @@ When the user is authenticated and `simple` is not set this returns something li "wiki_size" : 0, "lfs_objects_size": 0, "job_artifacts_size": 0, + "pipeline_artifacts_size": 0, "packages_size": 0, - "snippets_size": 0 + "snippets_size": 0, + "uploads_size": 0 }, "container_registry_image_prefix": "registry.example.com/brightbox/puppet", "_links": { @@ -460,6 +466,7 @@ GET /users/:user_id/projects "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "marked_for_deletion_at": "2020-04-03", // Deprecated and will be removed in API v5 in favor of marked_for_deletion_on "marked_for_deletion_on": "2020-04-03", "statistics": { @@ -469,8 +476,10 @@ GET /users/:user_id/projects "wiki_size" : 0, "lfs_objects_size": 0, "job_artifacts_size": 0, + "pipeline_artifacts_size": 0, "packages_size": 0, - "snippets_size": 0 + "snippets_size": 0, + "uploads_size": 0 }, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-client", "_links": { @@ -574,6 +583,7 @@ GET /users/:user_id/projects "service_desk_address": null, "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "statistics": { "commit_count": 12, "storage_size": 2066080, @@ -581,8 +591,10 @@ GET /users/:user_id/projects "wiki_size" : 0, "lfs_objects_size": 0, "job_artifacts_size": 0, + "pipeline_artifacts_size": 0, "packages_size": 0, - "snippets_size": 0 + "snippets_size": 0, + "uploads_size": 0 }, "container_registry_image_prefix": "registry.example.com/brightbox/puppet", "_links": { @@ -698,12 +710,17 @@ Example response: "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "statistics": { "commit_count": 37, "storage_size": 1038090, "repository_size": 1038090, "lfs_objects_size": 0, "job_artifacts_size": 0 + "pipeline_artifacts_size": 0, + "packages_size": 0, + "snippets_size": 0, + "uploads_size": 0 }, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-client", "_links": { @@ -805,12 +822,17 @@ Example response: "service_desk_address": null, "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "statistics": { "commit_count": 12, "storage_size": 2066080, "repository_size": 2066080, "lfs_objects_size": 0, "job_artifacts_size": 0 + "pipeline_artifacts_size": 0, + "packages_size": 0, + "snippets_size": 0, + "uploads_size": 0 }, "container_registry_image_prefix": "registry.example.com/brightbox/puppet", "_links": { @@ -968,6 +990,7 @@ GET /projects/:id "service_desk_address": null, "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "marked_for_deletion_at": "2020-04-03", // Deprecated and will be removed in API v5 in favor of marked_for_deletion_on "marked_for_deletion_on": "2020-04-03", "compliance_frameworks": [ "sox" ], @@ -978,8 +1001,10 @@ GET /projects/:id "wiki_size" : 0, "lfs_objects_size": 0, "job_artifacts_size": 0, + "pipeline_artifacts_size": 0, "packages_size": 0, - "snippets_size": 0 + "snippets_size": 0, + "uploads_size": 0 }, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-client", "_links": { @@ -1278,6 +1303,7 @@ POST /projects/user/:user_id | `issues_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | | `jobs_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | | `lfs_enabled` | boolean | **{dotted-circle}** No | Enable LFS. | +| `merge_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | | `merge_method` | string | **{dotted-circle}** No | Set the [merge method](#project-merge-method) used. | | `merge_requests_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | @@ -1355,6 +1381,7 @@ PUT /projects/:id | `issues_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable issues for this project. Use `issues_access_level` instead. | | `jobs_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable jobs for this project. Use `builds_access_level` instead. | | `lfs_enabled` | boolean | **{dotted-circle}** No | Enable LFS. | +| `merge_commit_template` | string | **{dotted-circle}** No | [Template](../user/project/merge_requests/commit_templates.md) used to create merge commit message in merge requests. _([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5.)_ | | `merge_method` | string | **{dotted-circle}** No | Set the [merge method](#project-merge-method) used. | | `merge_requests_access_level` | string | **{dotted-circle}** No | One of `disabled`, `private`, or `enabled`. | | `merge_requests_enabled` | boolean | **{dotted-circle}** No | _(Deprecated)_ Enable merge requests for this project. Use `merge_requests_access_level` instead. | @@ -1513,6 +1540,7 @@ Example responses: "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-project-site", "_links": { "self": "http://example.com/api/v4/projects", @@ -1614,6 +1642,7 @@ Example response: "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-project-site", "_links": { "self": "http://example.com/api/v4/projects", @@ -1713,6 +1742,7 @@ Example response: "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-project-site", "_links": { "self": "http://example.com/api/v4/projects", @@ -1906,6 +1936,7 @@ Example response: "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-project-site", "_links": { "self": "http://example.com/api/v4/projects", @@ -2026,6 +2057,7 @@ Example response: "squash_option": "default_on", "autoclose_referenced_issues": true, "suggestion_commit_message": null, + "merge_commit_template": null, "container_registry_image_prefix": "registry.example.com/diaspora/diaspora-project-site", "_links": { "self": "http://example.com/api/v4/projects", @@ -2652,6 +2684,7 @@ Example response: "merge_method": "merge", "squash_option": "default_on", "suggestion_commit_message": null, + "merge_commit_template": null, "auto_devops_enabled": true, "auto_devops_deploy_strategy": "continuous", "autoclose_referenced_issues": true, @@ -2673,6 +2706,10 @@ Read more in the [Project import/export](project_import_export.md) documentation Read more in the [Project members](members.md) documentation. +## Project vulnerabilities + +Read more in the [Project vulnerabilities](project_vulnerabilities.md) documentation. + ## Configure pull mirroring for a project **(PREMIUM)** > - Introduced in GitLab 11. diff --git a/doc/api/releases/index.md b/doc/api/releases/index.md index ded47b24c12..c253358f01f 100644 --- a/doc/api/releases/index.md +++ b/doc/api/releases/index.md @@ -26,6 +26,8 @@ For authentication, the Releases API accepts either: ## List Releases +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72448) to allow for `JOB-TOKEN` in GitLab 14.5. + Paginated list of Releases, sorted by `released_at`. ```plaintext @@ -231,6 +233,8 @@ Example response: ## Get a Release by a tag name +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72448) to allow for `JOB-TOKEN` in GitLab 14.5. + Get a Release for the given tag. ```plaintext @@ -508,7 +512,8 @@ adding milestones for ancestor groups raises an error. ## Collect release evidence **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/199065) in GitLab 12.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/199065) in GitLab 12.10. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72448) to allow for `JOB-TOKEN` in GitLab 14.5. Create Evidence for an existing Release. @@ -535,6 +540,8 @@ Example response: ## Update a release +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72448) to allow for `JOB-TOKEN` in GitLab 14.5. + Update a release. Developer level access to the project is required to update a release. ```plaintext @@ -642,6 +649,8 @@ Example response: ## Delete a Release +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72448) to allow for `JOB-TOKEN` in GitLab 14.5. + Delete a release. Deleting a release doesn't delete the associated tag. Maintainer level access to the project is required to delete a release. ```plaintext diff --git a/doc/api/repository_files.md b/doc/api/repository_files.md index 4fb1a94e294..cc210eacd49 100644 --- a/doc/api/repository_files.md +++ b/doc/api/repository_files.md @@ -185,7 +185,7 @@ POST /projects/:id/repository/files/:file_path ```shell curl --request POST --header 'PRIVATE-TOKEN: <your_access_token>' \ --header "Content-Type: application/json" \ - --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", \ + --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", "content": "some content", "commit_message": "create a new file"}' \ "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" ``` @@ -221,7 +221,7 @@ PUT /projects/:id/repository/files/:file_path ```shell curl --request PUT --header 'PRIVATE-TOKEN: <your_access_token>' \ --header "Content-Type: application/json" \ - --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", \ + --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", "content": "some content", "commit_message": "update file"}' \ "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" ``` @@ -268,7 +268,7 @@ DELETE /projects/:id/repository/files/:file_path ```shell curl --request DELETE --header 'PRIVATE-TOKEN: <your_access_token>' \ --header "Content-Type: application/json" \ - --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", \ + --data '{"branch": "master", "author_email": "author@example.com", "author_name": "Firstname Lastname", "commit_message": "delete file"}' \ "https://gitlab.example.com/api/v4/projects/13083/repository/files/app%2Fproject%2Erb" ``` diff --git a/doc/api/settings.md b/doc/api/settings.md index 7b8778973f2..dd32c882860 100644 --- a/doc/api/settings.md +++ b/doc/api/settings.md @@ -85,7 +85,7 @@ Example response: "raw_blob_request_limit": 300, "wiki_page_max_content_bytes": 52428800, "require_admin_approval_after_user_signup": false, - "personal_access_token_prefix": "GL-", + "personal_access_token_prefix": "glpat-", "rate_limiting_response_text": null, "keep_latest_artifact": true, "admin_mode": false, @@ -187,7 +187,7 @@ Example response: "raw_blob_request_limit": 300, "wiki_page_max_content_bytes": 52428800, "require_admin_approval_after_user_signup": false, - "personal_access_token_prefix": "GL-", + "personal_access_token_prefix": "glpat-", "rate_limiting_response_text": null, "keep_latest_artifact": true, "admin_mode": false, @@ -411,11 +411,11 @@ listed in the descriptions of the relevant settings. | `terminal_max_session_time` | integer | no | Maximum time for web terminal websocket connection (in seconds). Set to `0` for unlimited time. | | `terms` | text | required by: `enforce_terms` | (**Required by:** `enforce_terms`) Markdown content for the ToS. | | `throttle_authenticated_api_enabled` | boolean | no | (**If enabled, requires:** `throttle_authenticated_api_period_in_seconds` and `throttle_authenticated_api_requests_per_period`) Enable authenticated API request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | -| `throttle_authenticated_api_period_in_seconds` | integer | required by:<br>`throttle_authenticated_api_enabled` | Rate limit period in seconds. | -| `throttle_authenticated_api_requests_per_period` | integer | required by:<br>`throttle_authenticated_api_enabled` | Max requests per period per user. | +| `throttle_authenticated_api_period_in_seconds` | integer | required by:<br>`throttle_authenticated_api_enabled` | Rate limit period (in seconds). | +| `throttle_authenticated_api_requests_per_period` | integer | required by:<br>`throttle_authenticated_api_enabled` | Maximum requests per period per user. | | `throttle_authenticated_web_enabled` | boolean | no | (**If enabled, requires:** `throttle_authenticated_web_period_in_seconds` and `throttle_authenticated_web_requests_per_period`) Enable authenticated web request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | -| `throttle_authenticated_web_period_in_seconds` | integer | required by:<br>`throttle_authenticated_web_enabled` | Rate limit period in seconds. | -| `throttle_authenticated_web_requests_per_period` | integer | required by:<br>`throttle_authenticated_web_enabled` | Max requests per period per user. | +| `throttle_authenticated_web_period_in_seconds` | integer | required by:<br>`throttle_authenticated_web_enabled` | Rate limit period (in seconds). | +| `throttle_authenticated_web_requests_per_period` | integer | required by:<br>`throttle_authenticated_web_enabled` | Maximum requests per period per user. | | `throttle_unauthenticated_enabled` | boolean | no | ([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/335300) in GitLab 14.3. Use `throttle_unauthenticated_web_enabled` or `throttle_unauthenticated_api_enabled` instead.) (**If enabled, requires:** `throttle_unauthenticated_period_in_seconds` and `throttle_unauthenticated_requests_per_period`) Enable unauthenticated web request rate limit. Helps reduce request volume (for example, from crawlers or abusive bots). | | `throttle_unauthenticated_period_in_seconds` | integer | required by:<br>`throttle_unauthenticated_enabled` | ([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/335300) in GitLab 14.3. Use `throttle_unauthenticated_web_period_in_seconds` or `throttle_unauthenticated_api_period_in_seconds` instead.) Rate limit period in seconds. | | `throttle_unauthenticated_requests_per_period` | integer | required by:<br>`throttle_unauthenticated_enabled` | ([Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/335300) in GitLab 14.3. Use `throttle_unauthenticated_web_requests_per_period` or `throttle_unauthenticated_api_requests_per_period` instead.) Max requests per period per IP. | diff --git a/doc/api/sidekiq_metrics.md b/doc/api/sidekiq_metrics.md index bcb59a6dad3..2be2e71e551 100644 --- a/doc/api/sidekiq_metrics.md +++ b/doc/api/sidekiq_metrics.md @@ -6,8 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Sidekiq Metrics API **(FREE SELF)** -> Introduced in GitLab 8.9. - This API endpoint allows you to retrieve some information about the current state of Sidekiq, its jobs, queues, and processes. diff --git a/doc/api/todos.md b/doc/api/todos.md index 737bfb11da9..98f254f6620 100644 --- a/doc/api/todos.md +++ b/doc/api/todos.md @@ -27,7 +27,7 @@ Parameters: | `project_id` | integer | no | The ID of a project | | `group_id` | integer | no | The ID of a group | | `state` | string | no | The state of the to-do item. Can be either `pending` or `done` | -| `type` | string | no | The type of to-do item. Can be either `Issue`, `MergeRequest`, `DesignManagement::Design` or `AlertManagement::Alert` | +| `type` | string | no | The type of to-do item. Can be either `Issue`, `MergeRequest`, `Commit`, `Epic`, `DesignManagement::Design` or `AlertManagement::Alert` | ```shell curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/todos" diff --git a/doc/api/topics.md b/doc/api/topics.md new file mode 100644 index 00000000000..5e9e1b8fc12 --- /dev/null +++ b/doc/api/topics.md @@ -0,0 +1,190 @@ +--- +stage: Manage +group: Workspace +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Topics API **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340920) in GitLab 14.5. + +Interact with project topics using the REST API. + +## List topics + +Returns a list of project topics in the GitLab instance ordered by number of associated projects. + +```plaintext +GET /topics +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +| ---------- | ------- | ---------------------- | ----------- | +| `page` | integer | **{dotted-circle}** No | Page to retrieve. Defaults to `1`. | +| `per_page` | integer | **{dotted-circle}** No | Number of records to return per page. Defaults to `20`. | +| `search` | string | **{dotted-circle}** No | Search topics against their `name`. | + +Example request: + +```shell +curl "https://gitlab.example.com/api/v4/topics?search=git" +``` + +Example response: + +```json +[ + { + "id": 1, + "name": "GitLab", + "description": "GitLab is an open source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more.", + "total_projects_count": 1000, + "avatar_url": "http://www.gravatar.com/avatar/a0d477b3ea21970ce6ffcbb817b0b435?s=80&d=identicon" + }, + { + "id": 3, + "name": "Git", + "description": "Git is a free and open source distributed version control system designed to handle everything from small to very large projects with speed and efficiency.", + "total_projects_count": 900, + "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon" + }, + { + "id": 2, + "name": "Git LFS", + "description": null, + "total_projects_count": 300, + "avatar_url": null + } +] +``` + +## Get a topic + +Get a project topic by ID. + +```plaintext +GET /topics/:id +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +| --------- | ------- | ---------------------- | ------------------- | +| `id` | integer | **{check-circle}** Yes | ID of project topic | + +Example request: + +```shell +curl "https://gitlab.example.com/api/v4/topics/1" +``` + +Example response: + +```json +{ + "id": 1, + "name": "GitLab", + "description": "GitLab is an open source end-to-end software development platform with built-in version control, issue tracking, code review, CI/CD, and more.", + "total_projects_count": 1000, + "avatar_url": "http://www.gravatar.com/avatar/a0d477b3ea21970ce6ffcbb817b0b435?s=80&d=identicon" +} +``` + +## List projects assigned to a topic + +Use the [Projects API](projects.md#list-all-projects) to list all projects assigned to a specific topic. + +```plaintext +GET /projects?topic=<topic_name> +``` + +## Create a project topic + +Create a new project topic. Only available to administrators. + +```plaintext +POST /topics +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +| ------------- | ------- | ---------------------- | ----------- | +| `name` | string | **{check-circle}** Yes | Name | +| `avatar` | file | **{dotted-circle}** No | Avatar | +| `description` | string | **{dotted-circle}** No | Description | + +Example request: + +```shell +curl --request POST \ + --data "name=topic1" \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/topics" +``` + +Example response: + +```json +{ + "id": 1, + "name": "topic1", + "description": null, + "total_projects_count": 0, + "avatar_url": null +} +``` + +## Update a project topic + +Update a project topic. Only available to administrators. + +```plaintext +PUT /topics/:id +``` + +Supported attributes: + +| Attribute | Type | Required | Description | +| ------------- | ------- | ---------------------- | ------------------- | +| `id` | integer | **{check-circle}** Yes | ID of project topic | +| `avatar` | file | **{dotted-circle}** No | Avatar | +| `description` | string | **{dotted-circle}** No | Description | +| `name` | string | **{dotted-circle}** No | Name | + +Example request: + +```shell +curl --request PUT \ + --data "name=topic1" \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/topics/1" + + +Example response: + +```json +{ + "id": 1, + "name": "topic1", + "description": null, + "total_projects_count": 0, + "avatar_url": null +} +``` + +### Upload a topic avatar + +To upload an avatar file from your file system, use the `--form` argument. This argument causes +cURL to post data using the header `Content-Type: multipart/form-data`. The +`file=` parameter must point to a file on your file system and be preceded by +`@`. For example: + +```shell +curl --request PUT \ + --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/topics/1" \ + --form "avatar=@/tmp/example.png" +``` diff --git a/doc/architecture/blueprints/ci_scale/index.md b/doc/architecture/blueprints/ci_scale/index.md index 6afa13bf207..3e9fbc534d5 100644 --- a/doc/architecture/blueprints/ci_scale/index.md +++ b/doc/architecture/blueprints/ci_scale/index.md @@ -60,6 +60,9 @@ that have the same problem. Primary keys problem will be tackled by our Database Team. +Status: As of October 2021 the primary keys in CI tables have been migrated to +big integers. + ### The table is too large There is more than a billion rows in `ci_builds` table. We store more than 2 @@ -111,6 +114,12 @@ table that will accelerate SQL queries used to build queues](https://gitlab.com/gitlab-org/gitlab/-/issues/322766) and we want to explore them. +Status: the new architecture [has been implemented on GitLab.com](https://gitlab.com/groups/gitlab-org/-/epics/5909#note_680407908). + +The following epic tracks making it generally available: [Make the new pending +builds architecture generally available]( +https://gitlab.com/groups/gitlab-org/-/epics/6954). + ### Moving big amounts of data is challenging We store a significant amount of data in `ci_builds` table. Some of the columns @@ -127,6 +136,8 @@ columns, tables, partitions or database shards. Effort to improve background migrations will be owned by our Database Team. +Status: In progress. + ### Development velocity is negatively affected Team members and the wider community members are struggling to contribute the @@ -168,7 +179,12 @@ target](https://gitlab.com/groups/gitlab-org/-/epics/5745) epic. ## Status -In progress. +|-------------|--------------| +| Created at | 21.01.2021 | +| Approved at | 26.04.2021 | +| Updated at | 28.10.2021 | + +Status: In progress. ## Who @@ -180,7 +196,7 @@ Proposal: |------------------------------|-------------------------| | Author | Grzegorz Bizon | | Architecture Evolution Coach | Kamil Trzciński | -| Engineering Leader | Darby Frey | +| Engineering Leader | Cheryl Li | | Product Manager | Jackie Porter | | Domain Expert / Verify | Fabio Pitino | | Domain Expert / Database | Jose Finotto | @@ -190,7 +206,7 @@ DRIs: | Role | Who |------------------------------|------------------------| -| Leadership | Darby Frey | +| Leadership | Cheryl Li | | Product | Jackie Porter | | Engineering | Grzegorz Bizon | diff --git a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md index a8d87e5f967..53357220755 100644 --- a/doc/architecture/blueprints/consolidating_groups_and_projects/index.md +++ b/doc/architecture/blueprints/consolidating_groups_and_projects/index.md @@ -6,7 +6,7 @@ comments: false description: Consolidating groups and projects --- -# Consolidating Group and Project +# Consolidating Groups and Projects There are numerous features that exist exclusively within groups or projects. The boundary between group and project features used to be clear. @@ -127,6 +127,22 @@ The work required to establish `Namespace` as a container for our features is tracked under [Consolidate Groups and Projects](https://gitlab.com/groups/gitlab-org/-/epics/6473) epic. +## Migrating features to Namespaces + +The initial iteration will provide a framework to house features under `Namespaces`. Stage groups will eventually need to migrate their own features and functionality over to `Namespaces`. This may impact these features in unexpected ways. Therefore, to minimize UX debt and maintain product consistency, stage groups will have to consider a number of factors when migrating their features over to `Namespaces`: + +1. **Conceptual model**: What are the current and future state conceptual models of these features ([see object modeling for designers](https://hpadkisson.medium.com/object-modeling-for-designers-an-introduction-7871bdcf8baf))? These should be documented in Pajamas (example: [Merge Requests](https://design.gitlab.com/objects/merge-request)). +1. **Merge conflicts**: What inconsistencies are there across project, group, and admin levels? How might these be addressed? For an example of how we rationalized this for labels, please see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/338820). +1. **Inheritence & information flow**: How is information inherited across our container hierarchy currently? How might this be impacted if complying with the new [inheritence behavior](https://gitlab.com/gitlab-org/gitlab/-/issues/343316) framework? +1. **Settings**: Where can settings for this feature be found currently? How will these be impacted by `Namespaces`? +1. **Access**: Who can access this feature and is that impacted by the new container structure? Are there any role or privacy considerations? +1. **Tier**: Is there any tier functionality that is differentiated by projects and groups? +1. **Documentation**: Is the structure and content of documentation impacted by these changes at all? +1. **Solution proposal**: + - Think big: This analysis provides a great opportunity to zoom out and consider the feature UX as a whole. How could you make this feature lovable based on the new structure, inheritance, and capabilities afforded by `Namespaces`? Is there any UI which doesn't comply with Pajamas? + - Start small: What are the product changes that need to be made to assist with the migration? + - Move fast: Prioritise these solution ideas, document in issues, and create a roadmap for implementation. + ## Who Proposal: @@ -151,5 +167,10 @@ DRIs: | Product | Melissa Ushakov | | Leadership | Michelle Gill | | Engineering | Imre Farkas | +| Design | Nick Post | <!-- vale gitlab.Spelling = YES --> + +## Related topics + +- [Workspaces developer documentation](../../../development/workspaces/index.md) diff --git a/doc/architecture/blueprints/container_registry_metadata_database/index.md b/doc/architecture/blueprints/container_registry_metadata_database/index.md index f52635baf7b..7bbaefb8e1e 100644 --- a/doc/architecture/blueprints/container_registry_metadata_database/index.md +++ b/doc/architecture/blueprints/container_registry_metadata_database/index.md @@ -193,13 +193,19 @@ PostgreSQL introduced significant improvements for partitioning in [version 12]( - Bulk load (`COPY`) now uses bulk inserts instead of inserting one row at a time; To leverage these features and performance improvements, we need to use PostgreSQL 12 from the start. - -GitLab currently ships with PostgreSQL 11 for self-managed instances. That is _likely_ to change in 14.0, with PostgreSQL 12 becoming the minimum required version, which will likely happen before the upgraded registry becomes available for self-managed instances. If not, self-managed instances will have two options: - -1. Administrators can manually provision and configure a separate PostgreSQL 12 database for the registry, allowing them to benefit from features provided by the new registry and its metadata database. -1. Continue to use the current registry without the database if online garbage collection is not a concern or provisioning a separate database is not possible. We will continue supporting the current version with security backports and bug fixes for the foreseeable feature. - -It's important to note that apart from online garbage collection, the metadata database's availability will unblock the implementation of many requested features for the GitLab Container Registry, which will only be available for instances using the new version backed by the metadata database. +GitLab 14.0 and later [ships with PostgreSQL 12](../../../administration/package_information/postgresql_versions.md) +for self-managed instances. Customers not able to upgrade to PostgreSQL 12 have two options: + +- Administrators can manually provision and configure a separate PostgreSQL 12 database for the + registry. This lets you benefit from the features provided by the new registry and its metadata + database. +- If online garbage collection isn't a concern or provisioning a separate database isn't possible, + continue to use the current registry without the database. GitLab supports the current version + with security backports and bug fixes. + +Apart from online garbage collection, the metadata database's availability unblocks the +implementation of many requested features for the GitLab Container Registry. These features are only +available for instances using the new version backed by the metadata database. ### Availability diff --git a/doc/architecture/blueprints/feature_flags_development/index.md b/doc/architecture/blueprints/feature_flags_development/index.md index a5e46d25921..94d52ea3474 100644 --- a/doc/architecture/blueprints/feature_flags_development/index.md +++ b/doc/architecture/blueprints/feature_flags_development/index.md @@ -29,7 +29,7 @@ The extensive usage of feature flags poses a few challenges We sometimes forget how our feature flags are configured or why we haven't yet removed the feature flag. - The usage of feature flags can also be confusing to people outside of - development that might not fully understand dependence of ~feature or ~bug + development that might not fully understand dependence of ~"type::feature" or ~"type::bug" fix on feature flag and how this feature flag is configured. Or if the feature should be announced as part of release post. - Maintaining feature flags poses additional challenge of having to manage diff --git a/doc/ci/caching/index.md b/doc/ci/caching/index.md index 29a03fe06ab..57cbf387115 100644 --- a/doc/ci/caching/index.md +++ b/doc/ci/caching/index.md @@ -134,7 +134,7 @@ job: ## Inherit global configuration, but override specific settings per job You can override cache settings without overwriting the global cache by using -[anchors](../yaml/index.md#anchors). For example, if you want to override the +[anchors](../yaml/yaml_optimization.md#anchors). For example, if you want to override the `policy` for one job: ```yaml diff --git a/doc/ci/cloud_deployment/index.md b/doc/ci/cloud_deployment/index.md index 408d68fb726..80e4d8fbe27 100644 --- a/doc/ci/cloud_deployment/index.md +++ b/doc/ci/cloud_deployment/index.md @@ -104,7 +104,7 @@ GitLab also provides [Docker images](https://gitlab.com/gitlab-org/cloud-deploy/ - Use `registry.gitlab.com/gitlab-org/cloud-deploy/aws-ecs:latest` to deploy your application to AWS ECS. Before getting started with this process, you need a cluster on AWS ECS, as well as related -components, like an ECS service, ECS task definition, a database on AWS RDS, and so on. +components, like an ECS service, ECS task definition, a database on Amazon RDS, and so on. [Read more about AWS ECS](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/Welcome.html). The ECS task definition can be: diff --git a/doc/ci/directed_acyclic_graph/index.md b/doc/ci/directed_acyclic_graph/index.md index 11ba1e40b3e..f26a678962a 100644 --- a/doc/ci/directed_acyclic_graph/index.md +++ b/doc/ci/directed_acyclic_graph/index.md @@ -74,9 +74,9 @@ giving you powerful options for parallelization within your pipeline. ## Limitations A directed acyclic graph is a complicated feature, and as of the initial MVC there -are certain use cases that you may need to work around. For more information: +are certain use cases that you may need to work around. For more information, check the: -- [`needs` requirements and limitations](../yaml/index.md#requirements-and-limitations). +- [`needs` additional details](../yaml/index.md#needs). - Related epic [tracking planned improvements](https://gitlab.com/groups/gitlab-org/-/epics/1716). ## Needs Visualization diff --git a/doc/ci/docker/using_docker_build.md b/doc/ci/docker/using_docker_build.md index 9a4290ead4c..f6a6e892177 100644 --- a/doc/ci/docker/using_docker_build.md +++ b/doc/ci/docker/using_docker_build.md @@ -841,6 +841,8 @@ This issue occurs because Docker starts on TLS automatically. - If you are upgrading from v18.09 or earlier, read our [upgrade guide](https://about.gitlab.com/blog/2019/07/31/docker-in-docker-with-docker-19-dot-03/). +This error can also occur with the [Kubernetes executor](https://docs.gitlab.com/runner/executors/kubernetes.html#using-dockerdind) when attempts are made to access the Docker-in-Docker service before it has had time to fully start up. For a more detailed explanation, see [this issue](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27215). + ### Docker `no such host` error You may get an error that says diff --git a/doc/ci/docker/using_docker_images.md b/doc/ci/docker/using_docker_images.md index 79c23d73a68..def7703231c 100644 --- a/doc/ci/docker/using_docker_images.md +++ b/doc/ci/docker/using_docker_images.md @@ -132,16 +132,7 @@ For example, the following two definitions are equal: When a CI job runs in a Docker container, the `before_script`, `script`, and `after_script` commands run in the `/builds/<project-path>/` directory. Your image may have a different default `WORKDIR` defined. To move to your `WORKDIR`, save the `WORKDIR` as an environment variable so you can reference it in the container during the job's runtime. -### Available settings for `image` - -> Introduced in GitLab and GitLab Runner 9.4. - -| Setting | Required | Description | -|------------|----------| ----------- | -| `name` | Yes, when used with any other option. | Full name of the image. It should contain the registry part if needed. | -| `entrypoint` | No. | Command or script to execute as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | - -### Overriding the entrypoint of an image +### Override the entrypoint of an image > Introduced in GitLab and GitLab Runner 9.4. Read more about the [extended configuration options](../docker/using_docker_images.md#extended-docker-configuration-options). diff --git a/doc/ci/docker/using_kaniko.md b/doc/ci/docker/using_kaniko.md index 69c4557dcbe..ea3e81329d3 100644 --- a/doc/ci/docker/using_kaniko.md +++ b/doc/ci/docker/using_kaniko.md @@ -37,7 +37,7 @@ few important details: - The kaniko debug image is recommended (`gcr.io/kaniko-project/executor:debug`) because it has a shell, and a shell is required for an image to be used with GitLab CI/CD. -- The entrypoint needs to be [overridden](using_docker_images.md#overriding-the-entrypoint-of-an-image), +- The entrypoint needs to be [overridden](using_docker_images.md#override-the-entrypoint-of-an-image), otherwise the build script doesn't run. - A Docker `config.json` file needs to be created with the authentication information for the desired container registry. @@ -64,8 +64,12 @@ build: entrypoint: [""] script: - mkdir -p /kaniko/.docker - - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json - - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG + - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json + - >- + /kaniko/executor + --context "${CI_PROJECT_DIR}" + --dockerfile "${CI_PROJECT_DIR}/Dockerfile" + --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}" rules: - if: $CI_COMMIT_TAG ``` @@ -91,14 +95,19 @@ build: - mkdir -p /kaniko/.docker - |- KANIKOPROXYBUILDARGS="" - KANIKOCFG="{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" + KANIKOCFG="\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}" if [ "x${http_proxy}" != "x" -o "x${https_proxy}" != "x" ]; then KANIKOCFG="${KANIKOCFG}, \"proxies\": { \"default\": { \"httpProxy\": \"${http_proxy}\", \"httpsProxy\": \"${https_proxy}\", \"noProxy\": \"${no_proxy}\"}}" KANIKOPROXYBUILDARGS="--build-arg http_proxy=${http_proxy} --build-arg https_proxy=${https_proxy} --build-arg no_proxy=${no_proxy}" fi - KANIKOCFG="${KANIKOCFG} }" + KANIKOCFG="{ ${KANIKOCFG} }" echo "${KANIKOCFG}" > /kaniko/.docker/config.json - - /kaniko/executor --context $CI_PROJECT_DIR --dockerfile $CI_PROJECT_DIR/Dockerfile $KANIKOPROXYBUILDARGS --destination $CI_REGISTRY_IMAGE:$CI_COMMIT_TAG + - >- + /kaniko/executor + --context "${CI_PROJECT_DIR}" + --dockerfile "${CI_PROJECT_DIR}/Dockerfile" + "${KANIKOPROXYBUILDARGS}" + --destination "${CI_REGISTRY_IMAGE}:${CI_COMMIT_TAG}" rules: - if: $CI_COMMIT_TAG ``` @@ -120,7 +129,7 @@ store: ```yaml before_script: - mkdir -p /kaniko/.docker - - echo "{\"auths\":{\"$CI_REGISTRY\":{\"auth\":\"$(echo -n ${CI_REGISTRY_USER}:${CI_REGISTRY_PASSWORD} | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json + - echo "{\"auths\":{\"${CI_REGISTRY}\":{\"auth\":\"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"}}}" > /kaniko/.docker/config.json - | echo "-----BEGIN CERTIFICATE----- ... diff --git a/doc/ci/environments/index.md b/doc/ci/environments/index.md index ca81ad30317..87d7b9f9e30 100644 --- a/doc/ci/environments/index.md +++ b/doc/ci/environments/index.md @@ -21,13 +21,13 @@ GitLab: If you have a deployment service like [Kubernetes](../../user/infrastructure/clusters/index.md) associated with your project, you can use it to assist with your deployments. -You can even access a [web terminal](#web-terminals) for your environment from within GitLab. +You can even access a [web terminal](#web-terminals-deprecated) for your environment from within GitLab. ## View environments and deployments Prerequisites: -- You must have a minimum of [Reporter permission](../../user/permissions.md#project-members-permissions). +- You must have at least the Reporter [role](../../user/permissions.md#project-members-permissions). To view a list of environments and deployments: @@ -159,9 +159,9 @@ deploy_prod: environment: name: production url: https://example.com - when: manual rules: - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + when: manual ``` The `when: manual` action: @@ -171,9 +171,13 @@ The `when: manual` action: You can find the play button in the pipelines, environments, deployments, and jobs views. -## Configure Kubernetes deployments +## Configure Kubernetes deployments (DEPRECATED) -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27630) in GitLab 12.6. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. If you are deploying to a [Kubernetes cluster](../../user/infrastructure/clusters/index.md) associated with your project, you can configure these deployments from your @@ -308,6 +312,31 @@ Note the following: for these jobs. This ensures that runners can fetch the repository even after a feature branch is deleted. For more information, see [Ref Specs for Runners](../pipelines/index.md#ref-specs-for-runners). +## Track newly included merge requests per deployment + +GitLab can track newly included merge requests per deployment. +When a deployment succeeded, the system calculates commit-diffs between the latest deployment and the previous deployment. +This tracking information can be fetched via the [Deployment API](../../api/deployments.md#list-of-merge-requests-associated-with-a-deployment) +and displayed at a post-merge pipeline in [merge request pages](../../user/project/merge_requests/index.md). + +To activate this tracking, your environment must be configured in the following: + +- [Environment name](../yaml/index.md#environmentname) is not foldered with `/` (that is, top-level/long-lived environments), _OR_ +- [Environment tier](#deployment-tier-of-environments) is either `production` or `staging`. + +Here are the example setups of [`environment` keyword](../yaml/index.md#environment) in `.gitlab-ci.yml`: + +```yaml +# Trackable +environment: production +environment: production/aws +environment: development + +# Non Trackable +environment: review/$CI_COMMIT_REF_SLUG +environment: testing/aws +``` + ## Working with environments Once environments are configured, GitLab provides many features for working with them, @@ -631,7 +660,12 @@ It may take a minute or two for data to appear after initial deployment. Metric charts can be embedded in GitLab Flavored Markdown. See [Embedding Metrics in GitLab Flavored Markdown](../../operations/metrics/embed.md) for more details. -### Web terminals +### Web terminals (DEPRECATED) + +> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. If you deploy to your environments with the help of a deployment service (for example, the [Kubernetes integration](../../user/infrastructure/clusters/index.md)), GitLab can open @@ -680,6 +714,29 @@ fetch line: fetch = +refs/environments/*:refs/remotes/origin/environments/* ``` +### Archive Old Deployments + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73628) in GitLab 14.5. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available per project or for your entire instance, ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `deployments_archive`. On GitLab.com, this feature will be rolled out gradually. + +When a new deployment happens in your project, +GitLab creates [a special Git-ref to the deployment](#check-out-deployments-locally). +Since these Git-refs are populated from the remote GitLab repository, +you could find that some Git operations, such as `git-fetch` and `git-pull`, +become slower as the number of deployments in your project increases. + +To maintain the efficiency of your Git operations, GitLab keeps +only recent deployment refs (up to 50,000) and deletes the rest of the old deployment refs. +Archived deployments are still available, in the UI or by using the API, for auditing purposes. +Also, you can still fetch the deployed commit from the repository +with specifying the commit SHA (for example, `git checkout <deployment-sha>`), even after archive. + +NOTE: +GitLab preserves all commits as [`keep-around` refs](../../user/project/repository/reducing_the_repo_size_using_git.md) +so that deployed commits are not garbage collected, even if it's not referenced by the deployment refs. + ### Scope environments with specs > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/2112) in GitLab Premium 9.4. @@ -822,7 +879,9 @@ To ensure the `action: stop` can always run when needed, you can: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/21182) in GitLab 14.4. FLAG: -On self-managed GitLab, by default this bug fix is not available. To make it available per project or for your entire instance, ask an administrator to [enable the `surface_environment_creation_failure` flag](../../administration/feature_flags.md). On GitLab.com, this bug fix is not available, but will be rolled out shortly. +On self-managed GitLab, by default this bug fix is available. To hide the bug fix per project, +ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `surface_environment_creation_failure`. +On GitLab.com, this bug fix is available. If your project is configured to [create a dynamic environment](#create-a-dynamic-environment), you might encounter this error because the dynamically generated parameter can't be used for creating an environment. @@ -875,3 +934,19 @@ To fix this, use one of the following solutions: script: deploy review app environment: review/$CI_COMMIT_REF_SLUG ``` + +### Deployment refs are not found + +Starting from GitLab 14.5, GitLab [deletes old deployment refs](#archive-old-deployments) +to keep your Git repository performant. + +If you have to restore archived Git-refs, please ask an administrator of your self-managed GitLab instance +to execute the following command on Rails console: + +```ruby +Project.find_by_full_path(<your-project-full-path>).deployments.where(archived: true).each(&:create_ref) +``` + +Please note that GitLab could drop this support in the future for the performance concern. +You can open an issue in [GitLab Issue Tracker](https://gitlab.com/gitlab-org/gitlab/-/issues/new) +to discuss the behavior of this feature. diff --git a/doc/ci/environments/protected_environments.md b/doc/ci/environments/protected_environments.md index 47f93b03136..55b63dd090d 100644 --- a/doc/ci/environments/protected_environments.md +++ b/doc/ci/environments/protected_environments.md @@ -102,10 +102,9 @@ The group now has access and can be seen in the UI. ## Environment access by group membership -A user may be granted access to protected environments as part of -[group membership](../../user/group/index.md). Users with -[Reporter permissions](../../user/permissions.md), can only be granted access to -protected environments with this method. +A user may be granted access to protected environments as part of [group membership](../../user/group/index.md). Users +with the Reporter [role](../../user/permissions.md) can only be granted access to protected environments with this +method. ## Deployment branch access @@ -120,14 +119,14 @@ they have the following privileges: - [Stop an environment](index.md#stop-an-environment). - [Delete a stopped environment](index.md#delete-a-stopped-environment). -- [Create an environment terminal](index.md#web-terminals). +- [Create an environment terminal](index.md#web-terminals-deprecated). ## Deployment-only access to protected environments Users granted access to a protected environment, but not push or merge access to the branch deployed to it, are only granted access to deploy the environment. An individual in a -group with the Reporter permission, or in groups added to the project with Reporter permissions, -appears in the dropdown menu for deployment-only access. +group with the Reporter [role](../../user/permissions.md), or in groups added to the project with the Reporter +role, appears in the dropdown menu for deployment-only access. To add deployment-only access: @@ -136,7 +135,8 @@ To add deployment-only access: 1. Invite the group to be a project member. 1. Follow the steps in [Protecting Environments](#protecting-environments). -Note that deployment-only access is the only possible access level for groups with [Reporter permissions](../../user/permissions.md). +Note that deployment-only access is the only possible access level for groups with the Reporter +[role](../../user/permissions.md). ## Modifying and unprotecting environments diff --git a/doc/ci/git_submodules.md b/doc/ci/git_submodules.md index 0df653acca4..dc5faf0188e 100644 --- a/doc/ci/git_submodules.md +++ b/doc/ci/git_submodules.md @@ -59,3 +59,8 @@ To make submodules work correctly in CI/CD jobs: variables: GIT_SUBMODULE_STRATEGY: recursive ``` + +If you use the [`CI_JOB_TOKEN`](jobs/ci_job_token.md) to clone a submodule in a +pipeline job, the user executing the job must be assigned to a role that has +[permission](../user/permissions.md#gitlab-cicd-permissions) to trigger a pipeline +in the upstream submodule project. diff --git a/doc/ci/index.md b/doc/ci/index.md index 2f18bd28642..5dcb0bcb242 100644 --- a/doc/ci/index.md +++ b/doc/ci/index.md @@ -86,10 +86,10 @@ GitLab CI/CD features, grouped by DevOps stage, include: | [Browser Performance Testing](../user/project/merge_requests/browser_performance_testing.md) | Quickly determine the browser performance impact of pending code changes. | | [Load Performance Testing](../user/project/merge_requests/load_performance_testing.md) | Quickly determine the server performance impact of pending code changes. | | [CI services](services/index.md) | Link Docker containers with your base image. | -| [Code Quality](../user/project/merge_requests/code_quality.md) | Analyze your source code quality. | | [GitLab CI/CD for external repositories](ci_cd_for_external_repos/index.md) **(PREMIUM)** | Get the benefits of GitLab CI/CD combined with repositories in GitHub and Bitbucket Cloud. | | [Interactive Web Terminals](interactive_web_terminal/index.md) **(FREE SELF)** | Open an interactive web terminal to debug the running jobs. | -| [Unit test reports](unit_test_reports.md) | Identify script failures directly on merge requests. | +| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | +| [Unit test reports](unit_test_reports.md) | Identify test failures directly on merge requests. | | [Using Docker images](docker/using_docker_images.md) | Use GitLab and GitLab Runner with Docker to build and test applications. | |-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| | **Release** | | @@ -100,10 +100,10 @@ GitLab CI/CD features, grouped by DevOps stage, include: | [Feature Flags](../operations/feature_flags.md) | Deploy your features behind Feature Flags. | | [GitLab Pages](../user/project/pages/index.md) | Deploy static websites. | | [GitLab Releases](../user/project/releases/index.md) | Add release notes to Git tags. | -| [Review Apps](review_apps/index.md) | Configure GitLab CI/CD to preview code changes. | | [Cloud deployment](cloud_deployment/index.md) | Deploy your application to a main cloud provider. | |-------------------------------------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------| | **Secure** | | +| [Code Quality](../user/project/merge_requests/code_quality.md) | Analyze your source code quality. | | [Container Scanning](../user/application_security/container_scanning/index.md) **(ULTIMATE)** | Check your Docker containers for known vulnerabilities. | | [Dependency Scanning](../user/application_security/dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | | [License Compliance](../user/compliance/license_compliance/index.md) **(ULTIMATE)** | Search your project dependencies for their licenses. | @@ -129,13 +129,17 @@ See also: - [Enable or disable GitLab CI/CD in a project](enable_or_disable_ci.md). -## References +## Related topics Learn more about GitLab CI/CD: - [Why you might choose GitLab CI/CD](https://about.gitlab.com/blog/2016/10/17/gitlab-ci-oohlala/). - [Reasons you might migrate from another platform](https://about.gitlab.com/blog/2016/07/22/building-our-web-app-on-gitlab-ci/). - [5 Teams that made the switch to GitLab CI/CD](https://about.gitlab.com/blog/2019/04/25/5-teams-that-made-the-switch-to-gitlab-ci-cd/) +- If you use VS Code to edit your GitLab CI/CD configuration, the + [GitLab Workflow VS Code extension](../user/project/repository/vscode.md) helps you + [validate your configuration](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#validate-gitlab-ci-configuration) + and [view your pipeline status](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#information-about-your-branch-pipelines-mr-closing-issue). See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJaIOzCX4Vqg3dlwfELC3u2jEeCBbDk) presentation. @@ -144,6 +148,10 @@ See also the [Why CI/CD?](https://docs.google.com/presentation/d/1OGgk2Tcxbpl7DJ As GitLab CI/CD has evolved, certain breaking changes have been necessary. +#### 14.0 + +- No breaking changes. + #### 13.0 - [Remove Backported `os.Expand`](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/4915). diff --git a/doc/ci/jobs/ci_job_token.md b/doc/ci/jobs/ci_job_token.md index 308f38b22b7..b6a3011a3d6 100644 --- a/doc/ci/jobs/ci_job_token.md +++ b/doc/ci/jobs/ci_job_token.md @@ -20,7 +20,7 @@ You can use a GitLab CI/CD job token to authenticate with specific API endpoints - [Get job artifacts](../../api/job_artifacts.md#get-job-artifacts). - [Get job token's job](../../api/jobs.md#get-job-tokens-job). - [Pipeline triggers](../../api/pipeline_triggers.md), using the `token=` parameter. -- [Release creation](../../api/releases/index.md#create-a-release). +- [Releases](../../api/releases/index.md). - [Terraform plan](../../user/infrastructure/index.md). The token has the same permissions to access the API as the user that executes the diff --git a/doc/ci/jobs/index.md b/doc/ci/jobs/index.md index cb3b11ebf99..104badb782c 100644 --- a/doc/ci/jobs/index.md +++ b/doc/ci/jobs/index.md @@ -82,6 +82,24 @@ For example:  +## Job name limitations + +You can't use these keywords as job names: + +- `image` +- `services` +- `stages` +- `types` +- `before_script` +- `after_script` +- `variables` +- `cache` +- `include` + +Job names must be 255 characters or less. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342800) +in GitLab 14.5, [with a feature flag](../../administration/feature_flags.md) named `ci_validate_job_length`. +Enabled by default. To disable it, ask an administrator to [disable the feature flag](../../administration/feature_flags.md). + ## Group jobs in a pipeline If you have many similar jobs, your [pipeline graph](../pipelines/index.md#visualize-pipelines) becomes long and hard @@ -139,6 +157,89 @@ In [GitLab 13.8 and earlier](https://gitlab.com/gitlab-org/gitlab/-/merge_reques the regular expression is `\d+[\s:\/\\]+\d+\s*`. [Feature flag](../../user/feature_flags.md) removed in [GitLab 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/322080). +## Hide jobs + +To temporarily disable a job without deleting it from the configuration +file: + +- Comment out the job's configuration: + + ```yaml + # hidden_job: + # script: + # - run test + ``` + +- Start the job name with a dot (`.`) and it is not processed by GitLab CI/CD: + + ```yaml + .hidden_job: + script: + - run test + ``` + +You can use hidden jobs that start with `.` as templates for reusable configuration with: + +- The [`extends` keyword](../yaml/index.md#extends). +- [YAML anchors](../yaml/yaml_optimization.md#anchors). + +## Control the inheritance of default keywords and global variables + +You can control the inheritance of: + +- [default keywords](../yaml/index.md#default) with [`inherit:default`](../yaml/index.md#inheritdefault). +- [global variables](../yaml/index.md#default) with [`inherit:variables`](../yaml/index.md#inheritvariables). + +For example: + +```yaml +default: + image: 'ruby:2.4' + before_script: + - echo Hello World + +variables: + DOMAIN: example.com + WEBHOOK_URL: https://my-webhook.example.com + +rubocop: + inherit: + default: false + variables: false + script: bundle exec rubocop + +rspec: + inherit: + default: [image] + variables: [WEBHOOK_URL] + script: bundle exec rspec + +capybara: + inherit: + variables: false + script: bundle exec capybara + +karma: + inherit: + default: true + variables: [DOMAIN] + script: karma +``` + +In this example: + +- `rubocop`: + - inherits: Nothing. +- `rspec`: + - inherits: the default `image` and the `WEBHOOK_URL` variable. + - does **not** inherit: the default `before_script` and the `DOMAIN` variable. +- `capybara`: + - inherits: the default `before_script` and `image`. + - does **not** inherit: the `DOMAIN` and `WEBHOOK_URL` variables. +- `karma`: + - inherits: the default `image` and `before_script`, and the `DOMAIN` variable. + - does **not** inherit: `WEBHOOK_URL` variable. + ## Specifying variables when running manual jobs > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30485) in GitLab 12.2. diff --git a/doc/ci/jobs/job_control.md b/doc/ci/jobs/job_control.md index ad2bbbc883c..0f92ae5ca49 100644 --- a/doc/ci/jobs/job_control.md +++ b/doc/ci/jobs/job_control.md @@ -179,7 +179,7 @@ job: ``` You should not include both push and merge request pipelines in the same job without -[`workflow:rules` that prevent duplicate pipelines](../yaml/index.md#switch-between-branch-pipelines-and-merge-request-pipelines): +[`workflow:rules` that prevent duplicate pipelines](../yaml/workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines): ```yaml job: @@ -294,7 +294,7 @@ You can use the `$` character for both variables and paths. For example, if the > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322992) in GitLab 14.3. -Use [`!reference` tags](../yaml/index.md#reference-tags) to reuse rules in different +Use [`!reference` tags](../yaml/yaml_optimization.md#reference-tags) to reuse rules in different jobs. You can combine `!reference` rules with regular job-defined rules: ```yaml @@ -640,6 +640,106 @@ This job can no longer be scheduled to run automatically. You can, however, exec To start a delayed job immediately, select **Play** (**{play}**). Soon GitLab Runner starts the job. +## Parallelize large jobs + +To split a large job into multiple smaller jobs that run in parallel, use the +[`parallel`](../yaml/index.md#parallel) keyword in your `.gitlab-ci.yml` file. + +Different languages and test suites have different methods to enable parallelization. +For example, use [Semaphore Test Boosters](https://github.com/renderedtext/test-boosters) +and RSpec to run Ruby tests in parallel: + +```ruby +# Gemfile +source 'https://rubygems.org' + +gem 'rspec' +gem 'semaphore_test_boosters' +``` + +```yaml +test: + parallel: 3 + script: + - bundle + - bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL +``` + +You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec +job split into three separate jobs. + +WARNING: +Test Boosters reports usage statistics to the author. + +### Run a one-dimensional matrix of parallel jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) in GitLab 13.5. + +You can create a one-dimensional matrix of parallel jobs: + +```yaml +deploystacks: + stage: deploy + script: + - bin/deploy + parallel: + matrix: + - PROVIDER: [aws, ovh, gcp, vultr] +``` + +You can also [create a multi-dimensional matrix](../yaml/index.md#parallelmatrix). + +### Run a matrix of parallel trigger jobs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270957) in GitLab 13.10. + +You can run a [trigger](../yaml/index.md#trigger) job multiple times in parallel in a single pipeline, +but with different variable values for each instance of the job. + +```yaml +deploystacks: + stage: deploy + trigger: + include: path/to/child-pipeline.yml + parallel: + matrix: + - PROVIDER: aws + STACK: [monitoring, app1] + - PROVIDER: ovh + STACK: [monitoring, backup] + - PROVIDER: [gcp, vultr] + STACK: [data] +``` + +This example generates 6 parallel `deploystacks` trigger jobs, each with different values +for `PROVIDER` and `STACK`, and they create 6 different child pipelines with those variables. + +```plaintext +deploystacks: [aws, monitoring] +deploystacks: [aws, app1] +deploystacks: [ovh, monitoring] +deploystacks: [ovh, backup] +deploystacks: [gcp, data] +deploystacks: [vultr, data] +``` + +In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/239737), you can +use the variables defined in `parallel: matrix` with the [`tags`](../yaml/index.md#tags) keyword for +dynamic runner selection. + +```yaml +deploystacks: + stage: deploy + parallel: + matrix: + - PROVIDER: aws + STACK: [monitoring, app1] + - PROVIDER: gcp + STACK: [data] + tags: + - ${PROVIDER}-${STACK} +``` + ## Use predefined CI/CD variables to run jobs only in specific pipeline types You can use [predefined CI/CD variables](../variables/predefined_variables.md) to choose diff --git a/doc/ci/large_repositories/index.md b/doc/ci/large_repositories/index.md index 07b4cd80d6a..76e34df1f8c 100644 --- a/doc/ci/large_repositories/index.md +++ b/doc/ci/large_repositories/index.md @@ -250,12 +250,14 @@ concurrent = 4 This makes the cloning configuration to be part of the given runner and does not require us to update each `.gitlab-ci.yml`. -## Pre-clone step +## Git fetch caching or pre-clone step -> [An issue exists](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/463) to remove the need for this optimization. +For very active repositories with a large number of references and files, you can either (or both): -For very active repositories with a large number of references and files, you can also -optimize your CI jobs by seeding repository data with GitLab Runner's [`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section). - -See [our development documentation](../../development/pipelines.md#pre-clone-step) for -an overview of how we implemented this approach on GitLab.com for the main GitLab repository. +- Consider using the [Gitaly pack-objects cache](../../administration/gitaly/configure_gitaly.md#pack-objects-cache) instead of a + pre-clone step. This is easier to set up and it benefits all repositories on your GitLab server, unlike the pre-clone step that + must be configured per-repository. The pack-objects cache also automatically works for forks. On GitLab.com, where the pack-objects cache is + enabled on all Gitaly servers, we found that we no longer need a pre-clone step for `gitlab-org/gitlab` development. +- Optimize your CI/CD jobs by seeding repository data in a pre-clone step with the + [`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) of GitLab Runner. See the + [Runner Cloud for Linux](../runners/runner_cloud/linux_runner_cloud.md#pre-clone-script) for more details. diff --git a/doc/ci/lint.md b/doc/ci/lint.md index 45152e5a0df..e58907ee0bd 100644 --- a/doc/ci/lint.md +++ b/doc/ci/lint.md @@ -14,6 +14,9 @@ issues as well. To access the CI Lint tool, navigate to **CI/CD > Pipelines** or **CI/CD > Jobs** in your project and click **CI lint**. +If you use VS Code, you can also validate your CI/CD configuration with the +[GitLab Workflow VS Code extension](../user/project/repository/vscode.md). + ## Validate basic logic and syntax By default, the CI lint checks the syntax of your CI YAML configuration and also runs diff --git a/doc/ci/pipeline_editor/index.md b/doc/ci/pipeline_editor/index.md index 10a6d34b076..5be016aff40 100644 --- a/doc/ci/pipeline_editor/index.md +++ b/doc/ci/pipeline_editor/index.md @@ -82,8 +82,8 @@ where: - Configuration imported with [`include`](../yaml/index.md#include) is copied into the view. - Jobs that use [`extends`](../yaml/index.md#extends) display with the - [extended configuration merged into the job](../yaml/index.md#merge-details). -- YAML anchors are [replaced with the linked configuration](../yaml/index.md#anchors). + [extended configuration merged into the job](../yaml/yaml_optimization.md#merge-details). +- YAML anchors are [replaced with the linked configuration](../yaml/yaml_optimization.md#anchors). ## Commit changes to CI configuration diff --git a/doc/ci/pipelines/index.md b/doc/ci/pipelines/index.md index 69e974406e2..24e518b1f69 100644 --- a/doc/ci/pipelines/index.md +++ b/doc/ci/pipelines/index.md @@ -77,7 +77,7 @@ You can also configure specific aspects of your pipelines through the GitLab UI. - [Pipeline schedules](schedules.md). - [Custom CI/CD variables](../variables/index.md#custom-cicd-variables). -### Ref Specs for Runners +### Ref specs for runners When a runner picks a pipeline job, GitLab provides that job's metadata. This includes the [Git refspecs](https://git-scm.com/book/en/v2/Git-Internals-The-Refspec), which indicate which ref (branch, tag, and so on) and commit (SHA1) are checked out from your @@ -87,9 +87,9 @@ This table lists the refspecs injected for each pipeline type: | Pipeline type | Refspecs | |--------------- |---------------------------------------- | -| Pipeline for Branches | `+<sha>:refs/pipelines/<id>` and `+refs/heads/<name>:refs/remotes/origin/<name>` | -| pipeline for Tags | `+<sha>:refs/pipelines/<id>` and `+refs/tags/<name>:refs/tags/<name>` | -| [Pipeline for Merge Requests](../pipelines/merge_request_pipelines.md) | `+<sha>:refs/pipelines/<id>` | +| pipeline for branches | `+<sha>:refs/pipelines/<id>` and `+refs/heads/<name>:refs/remotes/origin/<name>` | +| pipeline for tags | `+<sha>:refs/pipelines/<id>` and `+refs/tags/<name>:refs/tags/<name>` | +| [pipeline for merge requests](../pipelines/merge_request_pipelines.md) | `+<sha>:refs/pipelines/<id>` | The refs `refs/heads/<name>` and `refs/tags/<name>` exist in your project repository. GitLab generates the special ref `refs/pipelines/<id>` during a @@ -127,6 +127,11 @@ you can filter the pipeline list by: [Starting in GitLab 14.2](https://gitlab.com/gitlab-org/gitlab/-/issues/26621), you can change the pipeline column to display the pipeline ID or the pipeline IID. +If you use VS Code to edit your GitLab CI/CD configuration, the +[GitLab Workflow VS Code extension](../../user/project/repository/vscode.md) helps you +[validate your configuration](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#validate-gitlab-ci-configuration) +and [view your pipeline status](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#information-about-your-branch-pipelines-mr-closing-issue). + ### Run a pipeline manually Pipelines can be manually executed, with predefined or manually-specified [variables](../variables/index.md). @@ -150,7 +155,7 @@ The pipeline now executes the jobs as configured. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. -You can use the [`value` and `description`](../yaml/index.md#prefill-variables-in-manual-pipelines) +You can use the [`value` and `description`](../yaml/index.md#variablesdescription) keywords to define [pipeline-level (global) variables](../variables/index.md#create-a-custom-cicd-variable-in-the-gitlab-ciyml-file) that are prefilled when running a pipeline manually. @@ -159,7 +164,7 @@ In pipelines triggered manually, the **Run pipelines** page displays all top-lev with a `description` and `value` defined in the `.gitlab-ci.yml` file. The values can then be modified if needed, which overrides the value for that single pipeline run. -The description is displayed below the variable. It can be used to explain what +The description is displayed next to the variable. It can be used to explain what the variable is used for, what the acceptable values are, and so on: ```yaml @@ -229,6 +234,15 @@ This functionality is only available: - For users with at least the Developer role. - If the stage contains [manual actions](#add-manual-interaction-to-your-pipeline). +### Skip a pipeline + +To push a commit without triggering a pipeline, add `[ci skip]` or `[skip ci]`, using any +capitalization, to your commit message. + +Alternatively, if you are using Git 2.10 or later, use the `ci.skip` [Git push option](../../user/project/push_options.md#push-options-for-gitlab-cicd). +The `ci.skip` push option does not skip merge request +pipelines. + ### Delete a pipeline > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/24851) in GitLab 12.7. @@ -391,6 +405,7 @@ be found when you go to: - The pipelines index page. - A single commit page. - A merge request page. +- The [pipeline editor](../pipeline_editor/index.md), [in GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/337514) and later. Pipeline mini graphs allow you to see all related jobs for a single commit and the net result of each stage of your pipeline. This allows you to quickly see what failed and diff --git a/doc/ci/pipelines/merge_request_pipelines.md b/doc/ci/pipelines/merge_request_pipelines.md index 5b40744aa79..119633d38e2 100644 --- a/doc/ci/pipelines/merge_request_pipelines.md +++ b/doc/ci/pipelines/merge_request_pipelines.md @@ -45,7 +45,7 @@ To do this, you can use [`rules`](#use-rules-to-run-pipelines-for-merge-requests ### Use `rules` to run pipelines for merge requests GitLab recommends that you use the `rules` keyword, which is available in -[`workflow:rules` templates](../yaml/index.md#workflowrules-templates). +[`workflow:rules` templates](../yaml/workflow.md#workflowrules-templates). ### Use `only` or `except` to run pipelines for merge requests @@ -199,7 +199,7 @@ If you are seeing two pipelines when using `only/except`, please see the caveats related to using `only/except` above (or, consider moving to `rules`). In [GitLab 13.7 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201845), -you can add `workflow:rules` to [switch from branch pipelines to merge request pipelines](../yaml/index.md#switch-between-branch-pipelines-and-merge-request-pipelines). +you can add `workflow:rules` to [switch from branch pipelines to merge request pipelines](../yaml/workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines). After a merge request is open on the branch, the pipeline switches to a merge request pipeline. ### Two pipelines created when pushing an invalid CI configuration file diff --git a/doc/ci/pipelines/multi_project_pipelines.md b/doc/ci/pipelines/multi_project_pipelines.md index 184961f4c95..30b3bc2e277 100644 --- a/doc/ci/pipelines/multi_project_pipelines.md +++ b/doc/ci/pipelines/multi_project_pipelines.md @@ -88,7 +88,7 @@ The keywords available for use in trigger jobs are: - [`only` and `except`](../yaml/index.md#only--except) - [`when`](../yaml/index.md#when) (only with a value of `on_success`, `on_failure`, or `always`) - [`extends`](../yaml/index.md#extends) -- [`needs`](../yaml/index.md#needs), but not [cross project artifact downloads with `needs`](../yaml/index.md#cross-project-artifact-downloads-with-needs) +- [`needs`](../yaml/index.md#needs), but not [`needs:project`](../yaml/index.md#needsproject) #### Specify a downstream pipeline branch @@ -154,7 +154,7 @@ trigger-downstream: trigger: my/project ``` -You can stop global variables from reaching the downstream pipeline by using the [`inherit` keyword](../yaml/index.md#inherit). +You can stop global variables from reaching the downstream pipeline by using the [`inherit:variables` keyword](../yaml/index.md#inheritvariables). In this example, the `MY_GLOBAL_VAR` variable is not available in the triggered pipeline: ```yaml @@ -190,7 +190,7 @@ the ones defined in the upstream project take precedence. #### Pass CI/CD variables to a downstream pipeline by using variable inheritance -You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](../variables/index.md#pass-an-environment-variable-to-another-job) and [cross project artifact downloads](../yaml/index.md#cross-project-artifact-downloads-with-needs). +You can pass variables to a downstream pipeline with [`dotenv` variable inheritance](../variables/index.md#pass-an-environment-variable-to-another-job) and [`needs:project`](../yaml/index.md#needsproject). In the upstream pipeline: @@ -254,21 +254,6 @@ trigger_job: strategy: depend ``` -#### Mirror status from upstream pipeline - -You can mirror the pipeline status from an upstream pipeline to a bridge job by -using the `needs:pipeline` keyword. The latest pipeline status from the default branch is -replicated to the bridge job. - -For example: - -```yaml -upstream_bridge: - stage: test - needs: - pipeline: other/project -``` - ### Create multi-project pipelines by using the API > [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/31573) to GitLab Free in 12.4. diff --git a/doc/ci/pipelines/parent_child_pipelines.md b/doc/ci/pipelines/parent_child_pipelines.md index e48728a904a..64f4160c963 100644 --- a/doc/ci/pipelines/parent_child_pipelines.md +++ b/doc/ci/pipelines/parent_child_pipelines.md @@ -81,7 +81,8 @@ microservice_a: trigger: include: - project: 'my-group/my-pipeline-library' - file: 'path/to/ci-config.yml' + ref: 'main' + file: '/path/to/child-pipeline.yml' ``` The maximum number of entries that are accepted for `trigger:include:` is three. @@ -98,7 +99,7 @@ microservice_a: strategy: depend ``` -## Merge Request child pipelines +## Merge request child pipelines To trigger a child pipeline as a [Merge Request Pipeline](merge_request_pipelines.md) we need to: @@ -149,7 +150,7 @@ microservice_a: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. Instead of running a child pipeline from a static YAML file, you can define a job that runs -your own script to generate a YAML file, which is then [used to trigger a child pipeline](../yaml/index.md#trigger-child-pipeline-with-generated-configuration-file). +your own script to generate a YAML file, which is then used to trigger a child pipeline. This technique can be very powerful in generating pipelines targeting content that changed or to build a matrix of targets and architectures. @@ -171,6 +172,31 @@ configuration for jobs, like scripts, that use the Windows runner would use `\`. In GitLab 12.9, the child pipeline could fail to be created in certain cases, causing the parent pipeline to fail. This is [resolved](https://gitlab.com/gitlab-org/gitlab/-/issues/209070) in GitLab 12.10. +### Dynamic child pipeline example + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. + +You can trigger a child pipeline from a [dynamically generated configuration file](../pipelines/parent_child_pipelines.md#dynamic-child-pipelines): + +```yaml +generate-config: + stage: build + script: generate-ci-config > generated-config.yml + artifacts: + paths: + - generated-config.yml + +child-pipeline: + stage: test + trigger: + include: + - artifact: generated-config.yml + job: generate-config +``` + +The `generated-config.yml` is extracted from the artifacts and used as the configuration +for triggering the child pipeline. + ## Nested child pipelines > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29651) in GitLab 13.4. diff --git a/doc/ci/pipelines/pipeline_artifacts.md b/doc/ci/pipelines/pipeline_artifacts.md index 55555571f97..b174b6af9f9 100644 --- a/doc/ci/pipelines/pipeline_artifacts.md +++ b/doc/ci/pipelines/pipeline_artifacts.md @@ -7,20 +7,24 @@ type: reference, howto # Pipeline artifacts **(FREE)** -Pipeline artifacts are files created by GitLab after a pipeline finishes. These are different than [job artifacts](job_artifacts.md) because they are not explicitly managed by the `.gitlab-ci.yml` definitions. +Pipeline artifacts are files created by GitLab after a pipeline finishes. Pipeline artifacts are +different to [job artifacts](job_artifacts.md) because they are not explicitly managed by +`.gitlab-ci.yml` definitions. -Pipeline artifacts are used by the [test coverage visualization feature](../../user/project/merge_requests/test_coverage_visualization.md) to collect coverage information. It uses the [`artifacts: reports`](../yaml/index.md#artifactsreports) CI/CD keyword. +Pipeline artifacts are used by the [test coverage visualization feature](../../user/project/merge_requests/test_coverage_visualization.md) +to collect coverage information. It uses the [`artifacts: reports`](../yaml/index.md#artifactsreports) CI/CD keyword. ## Storage -Pipeline artifacts are saved to disk or object storage. They count towards a project's [storage usage quota](../../user/usage_quotas.md#storage-usage-quota). The **Artifacts** on the Usage Quotas page is the sum of all job artifacts and pipeline artifacts. +Pipeline artifacts are saved to disk or object storage. They count towards a project's [storage usage quota](../../user/usage_quotas.md#storage-usage-quota). +The **Artifacts** on the Usage Quotas page is the sum of all job artifacts and pipeline artifacts. ## When pipeline artifacts are deleted -Pipeline artifacts are deleted either: +Pipeline artifacts from: -- Seven days after creation. -- After another pipeline runs successfully, if they are from the most recent successful - pipeline. +- The latest pipeline are kept forever. +- Pipelines superseded by a newer pipeline are deleted seven days after their creation date. -This deletion may take up to two days. +It can take up to two days for GitLab to delete pipeline artifacts from when they are due to be +deleted. diff --git a/doc/ci/pipelines/schedules.md b/doc/ci/pipelines/schedules.md index 90494a715ea..692460120fe 100644 --- a/doc/ci/pipelines/schedules.md +++ b/doc/ci/pipelines/schedules.md @@ -12,14 +12,16 @@ Pipelines are normally run based on certain conditions being met. For example, w Pipeline schedules can be used to also run [pipelines](index.md) at specific intervals. For example: -- Every month on the 22nd for a certain branch. -- Once every day. +- Every month on the 22nd (cron example: `0 0 22 * *`) for a certain branch. +- Every month on the 2nd Monday (cron example: `0 0 * * 1#2`). +- Every other Sunday at 0900 hours (cron example: `0 9 * * sun%2`). +- Once every day (cron example: `0 0 * * *`). + +Schedule timing is configured with cron notation, parsed by [Fugit](https://github.com/floraison/fugit). In addition to using the GitLab UI, pipeline schedules can be maintained using the [Pipeline schedules API](../../api/pipeline_schedules.md). -Schedule timing is configured with cron notation, parsed by [Fugit](https://github.com/floraison/fugit). - ## Prerequisites In order for a scheduled pipeline to be created successfully: diff --git a/doc/ci/pipelines/settings.md b/doc/ci/pipelines/settings.md index e14c1aa621f..a8ecb5e0d74 100644 --- a/doc/ci/pipelines/settings.md +++ b/doc/ci/pipelines/settings.md @@ -78,7 +78,7 @@ To avoid this scenario: 1. Select the **Skip outdated deployment jobs** checkbox. 1. Select **Save changes**. -Older deployment job are skipped when a new deployment starts. +Older deployment jobs are skipped when a new deployment starts. For more information, see [Deployment safety](../environments/deployment_safety.md). diff --git a/doc/ci/quick_start/index.md b/doc/ci/quick_start/index.md index 4006a1c9c3c..77a666c0cca 100644 --- a/doc/ci/quick_start/index.md +++ b/doc/ci/quick_start/index.md @@ -119,6 +119,11 @@ The pipeline starts when the commit is committed. #### `.gitlab-ci.yml` tips +- After you create your first `.gitlab-ci.yml` file, use the [pipeline editor](../pipeline_editor/index.md) + for all future edits to the file. With the pipeline editor, you can: + - Edit the pipeline configuration with automatic syntax highlighting and validation. + - View the [CI/CD configuration visualization](../pipeline_editor/index.md#visualize-ci-configuration), + a graphical representation of your `.gitlab-ci.yml` file. - If you want the runner to [use a Docker container to run the jobs](../docker/using_docker_images.md), edit the `.gitlab-ci.yml` file to include an image name: @@ -136,12 +141,8 @@ The pipeline starts when the commit is committed. Your application does not need to be built as a Docker container to run CI/CD jobs in Docker containers. -- To validate your `.gitlab-ci.yml` file, use the - [CI Lint tool](../lint.md), which is available in every project. -- You can also use [CI/CD configuration visualization](../pipeline_editor/index.md#visualize-ci-configuration) to - view a graphical representation of your `.gitlab-ci.yml` file. - Each job contains scripts and stages: - - The [`default`](../yaml/index.md#custom-default-keyword-values) keyword is for + - The [`default`](../yaml/index.md#default) keyword is for custom defaults, for example with [`before_script`](../yaml/index.md#before_script) and [`after_script`](../yaml/index.md#after_script). - [`stage`](../yaml/index.md#stage) describes the sequential execution of jobs. diff --git a/doc/ci/resource_groups/index.md b/doc/ci/resource_groups/index.md index 7de3643c0d4..25dacc9c437 100644 --- a/doc/ci/resource_groups/index.md +++ b/doc/ci/resource_groups/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w description: Control the job concurrency in GitLab CI/CD --- -# Resource Group **(FREE)** +# Resource group **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. @@ -13,7 +13,7 @@ By default, pipelines in GitLab CI/CD run in parallel. The parallelization is an the feedback loop in merge requests, however, there are some situations that you may want to limit the concurrency on deployment jobs to run them one by one. -Resource Group allows you to strategically control +Use resource groups to strategically control the concurrency of the jobs for optimizing your continuous deployments workflow with safety. ## Add a resource group @@ -139,9 +139,59 @@ Depending on the process mode of the resource group: - `deploy-1`, `deploy-2`, and `deploy-3` do not run in parallel. - `deploy-3` runs first, `deploy-2` runs second and `deploy-1` runs last. -## Pipeline-level concurrency control with Cross-Project/Parent-Child pipelines +## Pipeline-level concurrency control with cross-project/parent-child pipelines -See the how to [control the pipeline concurrency in cross-project pipelines](../yaml/index.md#pipeline-level-concurrency-control-with-cross-projectparent-child-pipelines). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39057) in GitLab 13.9. + +You can define `resource_group` for downstream pipelines that are sensitive to concurrent +executions. The [`trigger` keyword](../yaml/index.md#trigger) can trigger downstream pipelines and the +[`resource_group` keyword](../yaml/index.md#resource_group) can co-exist with it. `resource_group` is useful to control the +concurrency of deployment pipelines, while other jobs can continue to run concurrently. + +The following example has two pipeline configurations in a project. When a pipeline starts running, +non-sensitive jobs are executed first and aren't affected by concurrent executions in other +pipelines. However, GitLab ensures that there are no other deployment pipelines running before +triggering a deployment (child) pipeline. If other deployment pipelines are running, GitLab waits +until those pipelines finish before running another one. + +```yaml +# .gitlab-ci.yml (parent pipeline) + +build: + stage: build + script: echo "Building..." + +test: + stage: test + script: echo "Testing..." + +deploy: + stage: deploy + trigger: + include: deploy.gitlab-ci.yml + strategy: depend + resource_group: AWS-production +``` + +```yaml +# deploy.gitlab-ci.yml (child pipeline) + +stages: + - provision + - deploy + +provision: + stage: provision + script: echo "Provisioning..." + +deployment: + stage: deploy + script: echo "Deploying..." +``` + +You must define [`strategy: depend`](../yaml/index.md#triggerstrategy) +with the `trigger` keyword. This ensures that the lock isn't released until the downstream pipeline +finishes. ## API @@ -177,7 +227,7 @@ deploy: ``` In a parent pipeline, it runs the `test` job that subsequently runs a child pipeline, -and the [`strategy: depend` option](../yaml/index.md#linking-pipelines-with-triggerstrategy) makes the `test` job wait until the child pipeline has finished. +and the [`strategy: depend` option](../yaml/index.md#triggerstrategy) makes the `test` job wait until the child pipeline has finished. The parent pipeline runs the `deploy` job in the next stage, that requires a resource from the `production` resource group. If the process mode is `oldest_first`, it executes the jobs from the oldest pipelines, meaning the `deploy` job is going to be executed next. diff --git a/doc/ci/review_apps/index.md b/doc/ci/review_apps/index.md index 1b926d506c4..c67282643a4 100644 --- a/doc/ci/review_apps/index.md +++ b/doc/ci/review_apps/index.md @@ -1,6 +1,6 @@ --- -stage: Release -group: Release +stage: Verify +group: Testing info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- @@ -98,6 +98,7 @@ The following are example projects that demonstrate Review App configuration: - [NGINX](https://gitlab.com/gitlab-examples/review-apps-nginx). - [OpenShift](https://gitlab.com/gitlab-examples/review-apps-openshift). +- [HashiCorp Nomad](https://gitlab.com/gitlab-examples/review-apps-nomad). Other examples of Review Apps: diff --git a/doc/ci/runners/build_cloud/linux_build_cloud.md b/doc/ci/runners/build_cloud/linux_build_cloud.md index 6cdc93591a7..e8bad31c821 100644 --- a/doc/ci/runners/build_cloud/linux_build_cloud.md +++ b/doc/ci/runners/build_cloud/linux_build_cloud.md @@ -1,127 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../runner_cloud/linux_runner_cloud.md' +remove_date: '2022-02-05' --- -# Build Cloud runners for Linux **(FREE)** +This document was moved to [another location](../runner_cloud/linux_runner_cloud.md). -GitLab Build Cloud runners for Linux run in autoscale mode and are powered by Google Cloud Platform. - -Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job, thus maximizing security. These shared runners are available on GitLab.com. - -GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. - -All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, CoreOS and the latest Docker Engine -installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default -region of the VMs is US East1. -Each instance is used only for one job. This ensures that any sensitive data left on the system can't be accessed by other people's CI/CD jobs. - -The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. - -Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`), -**time out after 3 hours**, regardless of the timeout configured in a -project. Check the issues [#4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [#4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. - -Below are the runners' settings. - -| Setting | GitLab.com | Default | -| ----------- | ----------------- | ---------- | -| Executor | `docker+machine` | - | -| Default Docker image | `ruby:2.5` | - | -| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | - -## Pre-clone script - -Build Cloud runners for Linux provide a way to run commands in a CI -job before the runner attempts to run `git init` and `git fetch` to -download a GitLab repository. The -[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) -can be used for: - -- Seeding the build directory with repository data -- Sending a request to a server -- Downloading assets from a CDN -- Any other commands that must run before the `git init` - -To use this feature, define a [CI/CD variable](../../../ci/variables/index.md#custom-cicd-variables) called -`CI_PRE_CLONE_SCRIPT` that contains a bash script. - -[This example](../../../development/pipelines.md#pre-clone-step) -demonstrates how you might use a pre-clone step to seed the build -directory. - -NOTE: -The `CI_PRE_CLONE_SCRIPT` variable does not work on Windows runners. - -## `config.toml` - -The full contents of our `config.toml` are: - -NOTE: -Settings that are not public are shown as `X`. - -**Google Cloud Platform** - -```toml -concurrent = X -check_interval = 1 -metrics_server = "X" -sentry_dsn = "X" - -[[runners]] - name = "docker-auto-scale" - request_concurrency = X - url = "https://gitlab.com/" - token = "SHARED_RUNNER_TOKEN" - pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" - executor = "docker+machine" - environment = [ - "DOCKER_DRIVER=overlay2", - "DOCKER_TLS_CERTDIR=" - ] - limit = X - [runners.docker] - image = "ruby:2.5" - privileged = true - volumes = [ - "/certs/client", - "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. - ] - [runners.machine] - IdleCount = 50 - IdleTime = 3600 - MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. - MachineName = "srm-%s" - MachineDriver = "google" - MachineOptions = [ - "google-project=PROJECT", - "google-disk-size=25", - "google-machine-type=n1-standard-1", - "google-username=core", - "google-tags=gitlab-com,srm", - "google-use-internal-ip", - "google-zone=us-east1-d", - "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 - "google-machine-image=PROJECT/global/images/IMAGE", - "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. - "engine-opt=fixed-cidr-v6=fc00::/7", - "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 - ] - [[runners.machine.autoscaling]] - Periods = ["* * * * * sat,sun *"] - Timezone = "UTC" - IdleCount = 70 - IdleTime = 3600 - [[runners.machine.autoscaling]] - Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] - Timezone = "UTC" - IdleCount = 700 - IdleTime = 3600 - [runners.cache] - Type = "gcs" - Shared = true - [runners.cache.gcs] - CredentialsFile = "/path/to/file" - BucketName = "bucket-name" -``` +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/macos/environment.md b/doc/ci/runners/build_cloud/macos/environment.md index 8a2417186ae..aaef0d07098 100644 --- a/doc/ci/runners/build_cloud/macos/environment.md +++ b/doc/ci/runners/build_cloud/macos/environment.md @@ -1,43 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../../runner_cloud/macos/environment.md' +remove_date: '2022-02-05' --- -# VM instances and images for Build Cloud for macOS **(FREE)** +This document was moved to [another location](../../runner_cloud/macos/environment.md). -When you use the Build Cloud for macOS: - -- Each of your jobs runs in a newly provisioned VM, which is dedicated to the specific job. -- The VM is active only for the duration of the job and immediately deleted. - -## VM types - -The virtual machine where your job runs has `sudo` access with no password. -For the Beta, there is only one available machine type, `gbc-macos-large`. - -| Instance type | vCPUS | Memory (GB) | -| --------- | --- | ------- | -| `gbc-macos-large` | 4 | 10 | - -## VM images - -You can execute your build on one of the following images. -You specify this image in your `.gitlab-ci.yml` file. - -Each image is running a specific version of macOS and Xcode. - -| VM image | Included software | -|---------------------------|--------------------| -| macos-10.13-xcode-7 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | -| macos-10.13-xcode-8 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | -| macos-10.13-xcode-9 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | -| macos-10.14-xcode-10 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/mojave.yml> | -| macos-10.15-xcode-11 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/catalina.yml> | -| macos-11-xcode-12 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/big-sur.yml> | - -### Image update policy - -- Support for new macOS versions is planned. -- Additional details on the support policy and image update release process are documented - [in this project](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/55bf59c8fa88712960afff2bf6ecc5daa879a8f5/docs/overview.md#os-images). +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/macos_build_cloud.md b/doc/ci/runners/build_cloud/macos_build_cloud.md index 5d55462fb63..e478f93f34c 100644 --- a/doc/ci/runners/build_cloud/macos_build_cloud.md +++ b/doc/ci/runners/build_cloud/macos_build_cloud.md @@ -1,62 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../runner_cloud/macos_runner_cloud.md' +remove_date: '2022-02-05' --- -# Build Cloud runners for macOS (Beta) **(FREE SAAS)** +This document was moved to [another location](../runner_cloud/macos_runner_cloud.md). -The GitLab Build Cloud for macOS Beta provides on-demand runners integrated with GitLab SaaS [CI/CD](../../../ci/index.md). -Use these runners to build, test, and deploy apps for the Apple ecosystem (macOS, iOS, tvOS). You can take advantage -of all the capabilities of the GitLab single DevOps platform and not have to manage or operate a -build environment. - -Build Cloud runners for macOS are in [Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) -and shouldn't be relied upon for mission-critical production jobs. - -## Quickstart - -To start using Build Cloud for macOS Beta, you must submit an access request [issue](https://gitlab.com/gitlab-com/macos-buildcloud-runners-beta/-/issues/new?issuable_template=beta_access_request). After your -access has been granted and your build environment configured, you must configure your -`.gitlab-ci.yml` pipeline file: - -1. Add a `.gitlab-ci.yml` file to your project repository. -1. Specify the [image](macos/environment.md#vm-images) you want to use. -1. Commit a change to your repository. - -The runners automatically run your build. - -## Example `.gitlab-ci.yml` file - -The following sample `.gitlab-ci.yml` file shows how to start using the runners for macOS: - -```yaml -.macos_buildcloud_runners: - tags: - - shared-macos-amd64 - image: macos-11-xcode-12 - -stages: - - build - - test - -before_script: - - echo "started by ${GITLAB_USER_NAME}" - -build: - extends: - - .macos_buildcloud_runners - stage: build - script: - - echo "running scripts in the build job" - -test: - extends: - - .macos_buildcloud_runners - stage: test - script: - - echo "running scripts in the test job" -``` - -NOTE: -During the Beta period, the architecture of this solution will change. Rather than the jobs running on a specific VM instance, they will run on an ephemeral VM instance that is created by an autoscaling instance, known as the Runner Manager. We will notify all Beta participants of any downtime required to do this work. +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/build_cloud/windows_build_cloud.md b/doc/ci/runners/build_cloud/windows_build_cloud.md index e01de723055..8d57ecf27ed 100644 --- a/doc/ci/runners/build_cloud/windows_build_cloud.md +++ b/doc/ci/runners/build_cloud/windows_build_cloud.md @@ -1,155 +1,9 @@ --- -stage: Verify -group: Runner -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../runner_cloud/windows_runner_cloud.md' +remove_date: '2022-02-05' --- -# Build Cloud runners for Windows (beta) **(FREE)** +This document was moved to [another location](../runner_cloud/windows_runner_cloud.md). -GitLab Build Cloud runners for Windows are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) -and shouldn't be used for production workloads. - -During this beta period, the [shared runner pipeline quota](../../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) -applies for groups and projects in the same manner as Linux runners. This may -change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). - -Windows runners on GitLab.com autoscale by launching virtual machines on -the Google Cloud Platform. This solution uses an -[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) -developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). -Windows runners execute your CI/CD jobs on `n1-standard-2` instances with -2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in -the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). - -We want to keep iterating to get Windows runners in a stable state and -[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). -You can follow our work towards this goal in the -[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). - -## Configuration - -The full contents of our `config.toml` are: - -NOTE: -Settings that aren't public are shown as `X`. - -```toml -concurrent = X -check_interval = 3 - -[[runners]] - name = "windows-runner" - url = "https://gitlab.com/" - token = "TOKEN" - executor = "custom" - builds_dir = "C:\\GitLab-Runner\\builds" - cache_dir = "C:\\GitLab-Runner\\cache" - shell = "powershell" - [runners.custom] - config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] - prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] - run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] - cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" - cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] -``` - -The full contents of our `autoscaler/config.toml` are: - -```toml -Provider = "gcp" -Executor = "winrm" -OS = "windows" -LogLevel = "info" -LogFormat = "text" -LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" -VMTag = "windows" - -[GCP] - ServiceAccountFile = "PATH" - Project = "some-project-df9323" - Zone = "us-east1-c" - MachineType = "n1-standard-2" - Image = "IMAGE" - DiskSize = 50 - DiskType = "pd-standard" - Subnetwork = "default" - Network = "default" - Tags = ["TAGS"] - Username = "gitlab_runner" - -[WinRM] - MaximumTimeout = 3600 - ExecutionMaxRetries = 0 - -[ProviderCache] - Enabled = true - Directory = "C:\\GitLab-Runner\\autoscaler\\machines" -``` - -## Example `.gitlab-ci.yml` file - -Below is a sample `.gitlab-ci.yml` file that shows how to start using the runners for Windows: - -```yaml -.shared_windows_runners: - tags: - - shared-windows - - windows - - windows-1809 - -stages: - - build - - test - -before_script: - - Set-Variable -Name "time" -Value (date -Format "%H:%m") - - echo ${time} - - echo "started by ${GITLAB_USER_NAME}" - -build: - extends: - - .shared_windows_runners - stage: build - script: - - echo "running scripts in the build job" - -test: - extends: - - .shared_windows_runners - stage: test - script: - - echo "running scripts in the test job" -``` - -## Limitations and known issues - -- All the limitations mentioned in our [beta - definition](https://about.gitlab.com/handbook/product/#beta). -- The average provisioning time for a new Windows VM is 5 minutes. - This means that you may notice slower build start times - on the Windows runner fleet during the beta. In a future - release we intend to update the autoscaler to enable - the pre-provisioning of virtual machines. This is intended to significantly reduce - the time it takes to provision a VM on the Windows fleet. You can - follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). -- The Windows runner fleet may be unavailable occasionally - for maintenance or updates. -- The Windows runner virtual machine instances do not use the - GitLab Docker executor. This means that you can't specify - [`image`](../../../ci/yaml/index.md#image) or [`services`](../../../ci/yaml/index.md#services) in - your pipeline configuration. -- For the beta release, we have included a set of software packages in - the base VM image. If your CI job requires additional software that's - not included in this list, then you must add installation - commands to [`before_script`](../../../ci/yaml/index.md#before_script) or [`script`](../../../ci/yaml/index.md#script) to install the required - software. Note that each job runs on a new VM instance, so the - installation of additional software packages needs to be repeated for - each job in your pipeline. -- The job may stay in a pending state for longer than the - Linux runners. -- There is the possibility that we introduce breaking changes which will - require updates to pipelines that are using the Windows runner - fleet. +<!-- This redirect file can be deleted after 2022-02-05. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/ci/runners/configure_runners.md b/doc/ci/runners/configure_runners.md index 13897b934c5..9e30f4dbf4d 100644 --- a/doc/ci/runners/configure_runners.md +++ b/doc/ci/runners/configure_runners.md @@ -156,7 +156,7 @@ the GitLab instance. To determine this: 1. On the left sidebar, select **Overview > Runners**. 1. Find the runner in the table and view the **IP Address** column. - + ### Determine the IP address of a specific runner @@ -294,6 +294,9 @@ globally or for individual jobs: - [`GIT_FETCH_EXTRA_FLAGS`](#git-fetch-extra-flags) - [`GIT_DEPTH`](#shallow-cloning) (shallow cloning) - [`GIT_CLONE_PATH`](#custom-build-directories) (custom build directories) +- [`TRANSFER_METER_FREQUENCY`](#artifact-and-cache-settings) (artifact/cache meter update frequency) +- [`ARTIFACT_COMPRESSION_LEVEL`](#artifact-and-cache-settings) (artifact archiver compression level) +- [`CACHE_COMPRESSION_LEVEL`](#artifact-and-cache-settings) (cache archiver compression level) You can also use variables to configure how many times a runner [attempts certain stages of job execution](#job-stages-attempts). diff --git a/doc/ci/runners/img/shared_runner_ip_address_14_1.png b/doc/ci/runners/img/shared_runner_ip_address_14_1.png Binary files differdeleted file mode 100644 index d7eeeae7a3c..00000000000 --- a/doc/ci/runners/img/shared_runner_ip_address_14_1.png +++ /dev/null diff --git a/doc/ci/runners/img/shared_runner_ip_address_14_5.png b/doc/ci/runners/img/shared_runner_ip_address_14_5.png Binary files differnew file mode 100644 index 00000000000..9a579f69303 --- /dev/null +++ b/doc/ci/runners/img/shared_runner_ip_address_14_5.png diff --git a/doc/ci/runners/index.md b/doc/ci/runners/index.md index 9acca60c4b4..d408bc46609 100644 --- a/doc/ci/runners/index.md +++ b/doc/ci/runners/index.md @@ -5,12 +5,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference --- -# GitLab Build Cloud runners **(FREE)** +# GitLab Runner Cloud **(FREE)** If you are using self-managed GitLab or you want to use your own runners on GitLab.com, you can [install and configure your own runners](https://docs.gitlab.com/runner/install/). -If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on runners in the GitLab Build Cloud. +If you are using GitLab SaaS (GitLab.com), your CI jobs automatically run on runners in the GitLab Runner Cloud. No configuration is required. Your jobs can run on: - [Linux runners](build_cloud/linux_build_cloud.md). diff --git a/doc/ci/runners/runner_cloud/linux_runner_cloud.md b/doc/ci/runners/runner_cloud/linux_runner_cloud.md new file mode 100644 index 00000000000..d0fedfcabb2 --- /dev/null +++ b/doc/ci/runners/runner_cloud/linux_runner_cloud.md @@ -0,0 +1,186 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Runner Cloud for Linux **(FREE)** + +Runner Cloud runners for Linux run in autoscale mode and are powered by Google Cloud Platform. + +Autoscaling means reduced queue times to spin up CI/CD jobs, and isolated VMs for each job, thus maximizing security. These shared runners are available on GitLab.com. + +GitLab offers Ultimate tier capabilities and included CI/CD minutes per group per month for our [Open Source](https://about.gitlab.com/solutions/open-source/join/), [Education](https://about.gitlab.com/solutions/education/), and [Startups](https://about.gitlab.com/solutions/startups/) programs. For private projects, GitLab offers various [plans](https://about.gitlab.com/pricing/), starting with a Free tier. + +All your CI/CD jobs run on [n1-standard-1 instances](https://cloud.google.com/compute/docs/machine-types) with 3.75GB of RAM, Google COS and the latest Docker Engine +installed. Instances provide 1 vCPU and 25GB of HDD disk space. The default +region of the VMs is US East1. +Each instance is used only for one job. This ensures that any sensitive data left on the system can't be accessed by other people's CI/CD jobs. + +NOTE: +The final disk space your jobs can use will be less than 25GB. Some disk space allocated to the instance will be occupied by the operating system, the Docker image, and a copy of your cloned repository. + +The `gitlab-shared-runners-manager-X.gitlab.com` fleet of runners are dedicated for GitLab projects as well as community forks of them. They use a slightly larger machine type (n1-standard-2) and have a bigger SSD disk size. They don't run untagged jobs and unlike the general fleet of shared runners, the instances are re-used up to 40 times. + +Jobs handled by the shared runners on GitLab.com (`shared-runners-manager-X.gitlab.com`), +**time out after 3 hours**, regardless of the timeout configured in a +project. Check the issues [#4010](https://gitlab.com/gitlab-com/infrastructure/-/issues/4010) and [#4070](https://gitlab.com/gitlab-com/infrastructure/-/issues/4070) for the reference. + +Below are the runners' settings. + +| Setting | GitLab.com | Default | +| ----------- | ----------------- | ---------- | +| Executor | `docker+machine` | - | +| Default Docker image | `ruby:2.5` | - | +| `privileged` (run [Docker in Docker](https://hub.docker.com/_/docker/)) | `true` | `false` | + +These runners share a [distributed cache](https://docs.gitlab.com/runner/configuration/autoscale.html#distributed-runners-caching) through use of a Google Cloud Storage (GCS) bucket. Cache contents not updated within the last 14 days are automatically removed through use of an [object lifecycle management policy](https://cloud.google.com/storage/docs/lifecycle). + +## Pre-clone script + +Cloud runners for Linux provide a way to run commands in a CI +job before the runner attempts to run `git init` and `git fetch` to +download a GitLab repository. The +[`pre_clone_script`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-section) +can be used for: + +- Seeding the build directory with repository data +- Sending a request to a server +- Downloading assets from a CDN +- Any other commands that must run before the `git init` + +To use this feature, define a [CI/CD variable](../../../ci/variables/index.md#custom-cicd-variables) called +`CI_PRE_CLONE_SCRIPT` that contains a bash script. + +NOTE: +The `CI_PRE_CLONE_SCRIPT` variable does not work on Windows runners. + +### Pre-clone script example + +This example was used in the `gitlab-org/gitlab` project until November 2021. +The project no longer uses this optimization because the [pack-objects cache](../../../administration/gitaly/configure_gitaly.md#pack-objects-cache) +lets Gitaly serve the full CI/CD fetch traffic. See [Git fetch caching](../../../development/pipelines.md#git-fetch-caching). + +The `CI_PRE_CLONE_SCRIPT` was defined as a project CI/CD variable: + +```shell +( + echo "Downloading archived master..." + wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz + + if [ ! -f /tmp/gitlab.tar.gz ]; then + echo "Repository cache not available, cloning a new directory..." + exit + fi + + rm -rf $CI_PROJECT_DIR + echo "Extracting tarball into $CI_PROJECT_DIR..." + mkdir -p $CI_PROJECT_DIR + cd $CI_PROJECT_DIR + tar xzf /tmp/gitlab.tar.gz + rm -f /tmp/gitlab.tar.gz + chmod a+w $CI_PROJECT_DIR +) +``` + +The first step of the script downloads `gitlab-master.tar.gz` from Google Cloud Storage. +There was a [GitLab CI/CD job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/5fb40526c8c8aaafc5f92eab36d5bbddaca3893d/.gitlab/ci/cache-repo.gitlab-ci.yml) +that was responsible for keeping that archive up-to-date. Every two hours on a scheduled pipeline, +it did the following: + +1. Create a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. +1. Save the data as a `.tar.gz`. +1. Upload it into the Google Cloud Storage bucket. + +When a job ran with this configuration, the output looked similar to: + +```shell +$ eval "$CI_PRE_CLONE_SCRIPT" +Downloading archived master... +Extracting tarball into /builds/gitlab-org/gitlab... +Fetching changes... +Reinitialized existing Git repository in /builds/gitlab-org/gitlab/.git/ +``` + +The `Reinitialized existing Git repository` message shows that +the pre-clone step worked. The runner runs `git init`, which +overwrites the Git configuration with the appropriate settings to fetch +from the GitLab repository. + +`CI_REPO_CACHE_CREDENTIALS` must contain the Google Cloud service account +JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. + +Note that this bucket should be located in the same continent as the +runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). + +## `config.toml` + +The full contents of our `config.toml` are: + +NOTE: +Settings that are not public are shown as `X`. + +**Google Cloud Platform** + +```toml +concurrent = X +check_interval = 1 +metrics_server = "X" +sentry_dsn = "X" + +[[runners]] + name = "docker-auto-scale" + request_concurrency = X + url = "https://gitlab.com/" + token = "SHARED_RUNNER_TOKEN" + pre_clone_script = "eval \"$CI_PRE_CLONE_SCRIPT\"" + executor = "docker+machine" + environment = [ + "DOCKER_DRIVER=overlay2", + "DOCKER_TLS_CERTDIR=" + ] + limit = X + [runners.docker] + image = "ruby:2.5" + privileged = true + volumes = [ + "/certs/client", + "/dummy-sys-class-dmi-id:/sys/class/dmi/id:ro" # Make kaniko builds work on GCP. + ] + [runners.machine] + IdleCount = 50 + IdleTime = 3600 + MaxBuilds = 1 # For security reasons we delete the VM after job has finished so it's not reused. + MachineName = "srm-%s" + MachineDriver = "google" + MachineOptions = [ + "google-project=PROJECT", + "google-disk-size=25", + "google-machine-type=n1-standard-1", + "google-username=core", + "google-tags=gitlab-com,srm", + "google-use-internal-ip", + "google-zone=us-east1-d", + "engine-opt=mtu=1460", # Set MTU for container interface, for more information check https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3214#note_82892928 + "google-machine-image=PROJECT/global/images/IMAGE", + "engine-opt=ipv6", # This will create IPv6 interfaces in the containers. + "engine-opt=fixed-cidr-v6=fc00::/7", + "google-operation-backoff-initial-interval=2" # Custom flag from forked docker-machine, for more information check https://github.com/docker/machine/pull/4600 + ] + [[runners.machine.autoscaling]] + Periods = ["* * * * * sat,sun *"] + Timezone = "UTC" + IdleCount = 70 + IdleTime = 3600 + [[runners.machine.autoscaling]] + Periods = ["* 30-59 3 * * * *", "* 0-30 4 * * * *"] + Timezone = "UTC" + IdleCount = 700 + IdleTime = 3600 + [runners.cache] + Type = "gcs" + Shared = true + [runners.cache.gcs] + CredentialsFile = "/path/to/file" + BucketName = "bucket-name" +``` diff --git a/doc/ci/runners/runner_cloud/macos/environment.md b/doc/ci/runners/runner_cloud/macos/environment.md new file mode 100644 index 00000000000..ddefad775c1 --- /dev/null +++ b/doc/ci/runners/runner_cloud/macos/environment.md @@ -0,0 +1,43 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# VM instances and images for Runner Cloud for macOS **(FREE)** + +When you use the Runner Cloud for macOS: + +- Each of your jobs runs in a newly provisioned VM, which is dedicated to the specific job. +- The VM is active only for the duration of the job and immediately deleted. + +## VM types + +The virtual machine where your job runs has `sudo` access with no password. +For the Beta, there is only one available machine type, `gbc-macos-large`. + +| Instance type | vCPUS | Memory (GB) | +| --------- | --- | ------- | +| `gbc-macos-large` | 4 | 10 | + +## VM images + +You can execute your build on one of the following images. +You specify this image in your `.gitlab-ci.yml` file. + +Each image is running a specific version of macOS and Xcode. + +| VM image | Included software | +|---------------------------|--------------------| +| macos-10.13-xcode-7 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | +| macos-10.13-xcode-8 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | +| macos-10.13-xcode-9 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/high-sierra.yml> | +| macos-10.14-xcode-10 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/mojave.yml> | +| macos-10.15-xcode-11 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/catalina.yml> | +| macos-11-xcode-12 | <https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/main/toolchain/big-sur.yml> | + +### Image update policy + +- Support for new macOS versions is planned. +- Additional details on the support policy and image update release process are documented + [in this project](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/macstadium/orka/-/blob/55bf59c8fa88712960afff2bf6ecc5daa879a8f5/docs/overview.md#os-images). diff --git a/doc/ci/runners/runner_cloud/macos_runner_cloud.md b/doc/ci/runners/runner_cloud/macos_runner_cloud.md new file mode 100644 index 00000000000..332284fa8c1 --- /dev/null +++ b/doc/ci/runners/runner_cloud/macos_runner_cloud.md @@ -0,0 +1,62 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Runner Cloud for macOS (Beta) **(FREE SAAS)** + +The Runner Cloud for macOS Beta provides on-demand runners integrated with GitLab SaaS [CI/CD](../../../ci/index.md). +Use these runners to build, test, and deploy apps for the Apple ecosystem (macOS, iOS, tvOS). You can take advantage +of all the capabilities of the GitLab single DevOps platform and not have to manage or operate a +build environment. + +Cloud runners for macOS are in [Beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +and shouldn't be relied upon for mission-critical production jobs. + +## Quickstart + +To start using Runner Cloud for macOS Beta, you must submit an access request [issue](https://gitlab.com/gitlab-com/macos-buildcloud-runners-beta/-/issues/new?issuable_template=beta_access_request). After your +access has been granted and your build environment configured, you must configure your +`.gitlab-ci.yml` pipeline file: + +1. Add a `.gitlab-ci.yml` file to your project repository. +1. Specify the [image](macos/environment.md#vm-images) you want to use. +1. Commit a change to your repository. + +The runners automatically run your build. + +## Example `.gitlab-ci.yml` file + +The following sample `.gitlab-ci.yml` file shows how to start using the runners for macOS: + +```yaml +.macos_buildcloud_runners: + tags: + - shared-macos-amd64 + image: macos-11-xcode-12 + +stages: + - build + - test + +before_script: + - echo "started by ${GITLAB_USER_NAME}" + +build: + extends: + - .macos_buildcloud_runners + stage: build + script: + - echo "running scripts in the build job" + +test: + extends: + - .macos_buildcloud_runners + stage: test + script: + - echo "running scripts in the test job" +``` + +NOTE: +During the Beta period, the architecture of this solution will change. Rather than the jobs running on a specific VM instance, they will run on an ephemeral VM instance that is created by an autoscaling instance, known as the Runner Manager. We will notify all Beta participants of any downtime required to do this work. diff --git a/doc/ci/runners/runner_cloud/windows_runner_cloud.md b/doc/ci/runners/runner_cloud/windows_runner_cloud.md new file mode 100644 index 00000000000..ef4d4076c91 --- /dev/null +++ b/doc/ci/runners/runner_cloud/windows_runner_cloud.md @@ -0,0 +1,155 @@ +--- +stage: Verify +group: Runner +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Runner Cloud for Windows (beta) **(FREE)** + +Runner Cloud runners for Windows are in [beta](https://about.gitlab.com/handbook/product/gitlab-the-product/#beta) +and shouldn't be used for production workloads. + +During this beta period, the [shared runner pipeline quota](../../../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) +applies for groups and projects in the same manner as Linux runners. This may +change when the beta period ends, as discussed in this [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/30834). + +Windows runners on GitLab.com autoscale by launching virtual machines on +the Google Cloud Platform. This solution uses an +[autoscaling driver](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/tree/master/docs/readme.md) +developed by GitLab for the [custom executor](https://docs.gitlab.com/runner/executors/custom.html). +Windows runners execute your CI/CD jobs on `n1-standard-2` instances with +2 vCPUs and 7.5 GB RAM. You can find a full list of available Windows packages in +the [package documentation](https://gitlab.com/gitlab-org/ci-cd/shared-runners/images/gcp/windows-containers/blob/main/cookbooks/preinstalled-software/README.md). + +We want to keep iterating to get Windows runners in a stable state and +[generally available](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga). +You can follow our work towards this goal in the +[related epic](https://gitlab.com/groups/gitlab-org/-/epics/2162). + +## Configuration + +The full contents of our `config.toml` are: + +NOTE: +Settings that aren't public are shown as `X`. + +```toml +concurrent = X +check_interval = 3 + +[[runners]] + name = "windows-runner" + url = "https://gitlab.com/" + token = "TOKEN" + executor = "custom" + builds_dir = "C:\\GitLab-Runner\\builds" + cache_dir = "C:\\GitLab-Runner\\cache" + shell = "powershell" + [runners.custom] + config_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + config_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "config"] + prepare_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + prepare_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "prepare"] + run_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + run_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "run"] + cleanup_exec = "C:\\GitLab-Runner\\autoscaler\\autoscaler.exe" + cleanup_args = ["--config", "C:\\GitLab-Runner\\autoscaler\\config.toml", "custom", "cleanup"] +``` + +The full contents of our `autoscaler/config.toml` are: + +```toml +Provider = "gcp" +Executor = "winrm" +OS = "windows" +LogLevel = "info" +LogFormat = "text" +LogFile = "C:\\GitLab-Runner\\autoscaler\\autoscaler.log" +VMTag = "windows" + +[GCP] + ServiceAccountFile = "PATH" + Project = "some-project-df9323" + Zone = "us-east1-c" + MachineType = "n1-standard-2" + Image = "IMAGE" + DiskSize = 50 + DiskType = "pd-standard" + Subnetwork = "default" + Network = "default" + Tags = ["TAGS"] + Username = "gitlab_runner" + +[WinRM] + MaximumTimeout = 3600 + ExecutionMaxRetries = 0 + +[ProviderCache] + Enabled = true + Directory = "C:\\GitLab-Runner\\autoscaler\\machines" +``` + +## Example `.gitlab-ci.yml` file + +Below is a sample `.gitlab-ci.yml` file that shows how to start using the runners for Windows: + +```yaml +.shared_windows_runners: + tags: + - shared-windows + - windows + - windows-1809 + +stages: + - build + - test + +before_script: + - Set-Variable -Name "time" -Value (date -Format "%H:%m") + - echo ${time} + - echo "started by ${GITLAB_USER_NAME}" + +build: + extends: + - .shared_windows_runners + stage: build + script: + - echo "running scripts in the build job" + +test: + extends: + - .shared_windows_runners + stage: test + script: + - echo "running scripts in the test job" +``` + +## Limitations and known issues + +- All the limitations mentioned in our [beta + definition](https://about.gitlab.com/handbook/product/#beta). +- The average provisioning time for a new Windows VM is 5 minutes. + This means that you may notice slower build start times + on the Windows runner fleet during the beta. In a future + release we intend to update the autoscaler to enable + the pre-provisioning of virtual machines. This is intended to significantly reduce + the time it takes to provision a VM on the Windows fleet. You can + follow along in the [related issue](https://gitlab.com/gitlab-org/ci-cd/custom-executor-drivers/autoscaler/-/issues/32). +- The Windows runner fleet may be unavailable occasionally + for maintenance or updates. +- The Windows runner virtual machine instances do not use the + GitLab Docker executor. This means that you can't specify + [`image`](../../../ci/yaml/index.md#image) or [`services`](../../../ci/yaml/index.md#services) in + your pipeline configuration. +- For the beta release, we have included a set of software packages in + the base VM image. If your CI job requires additional software that's + not included in this list, then you must add installation + commands to [`before_script`](../../../ci/yaml/index.md#before_script) or [`script`](../../../ci/yaml/index.md#script) to install the required + software. Note that each job runs on a new VM instance, so the + installation of additional software packages needs to be repeated for + each job in your pipeline. +- The job may stay in a pending state for longer than the + Linux runners. +- There is the possibility that we introduce breaking changes which will + require updates to pipelines that are using the Windows runner + fleet. diff --git a/doc/ci/services/index.md b/doc/ci/services/index.md index 4114833d1c8..51df8c102f7 100644 --- a/doc/ci/services/index.md +++ b/doc/ci/services/index.md @@ -259,7 +259,8 @@ test: | `name` | yes, when used with any other option | 9.4 | Full name of the image to use. If the full image name includes a registry hostname, use the `alias` option to define a shorter service access name. For more information, see [Accessing the services](#accessing-the-services). | | `entrypoint` | no | 9.4 |Command or script to execute as the container's entrypoint. It's translated to Docker's `--entrypoint` option while creating the container. The syntax is similar to [`Dockerfile`'s `ENTRYPOINT`](https://docs.docker.com/engine/reference/builder/#entrypoint) directive, where each shell token is a separate string in the array. | | `command` | no | 9.4 |Command or script that should be used as the container's command. It's translated to arguments passed to Docker after the image's name. The syntax is similar to [`Dockerfile`'s `CMD`](https://docs.docker.com/engine/reference/builder/#cmd) directive, where each shell token is a separate string in the array. | -| `alias` (1) | no | 9.4 |Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | +| `alias` (1) | no | 9.4 | Additional alias that can be used to access the service from the job's container. Read [Accessing the services](#accessing-the-services) for more information. | +| `variables` | no | 14.5 | Additional environment variables that are passed exclusively to the service. The syntax is the same as [Job Variables](../variables/index.md). | (1) Alias support for the Kubernetes executor was [introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2229) in GitLab Runner 12.8, and is only available for Kubernetes version 1.7 or later. diff --git a/doc/ci/services/postgres.md b/doc/ci/services/postgres.md index 62838e04969..0bd43917cd1 100644 --- a/doc/ci/services/postgres.md +++ b/doc/ci/services/postgres.md @@ -37,9 +37,9 @@ And then configure your application to use the database, for example: ```yaml Host: postgres -User: $PG_USER -Password: $PG_PASSWORD -Database: $PG_DB +User: $POSTGRES_USER +Password: $POSTGRES_PASSWORD +Database: $POSTGRES_DB ``` If you're wondering why we used `postgres` for the `Host`, read more at diff --git a/doc/ci/test_cases/index.md b/doc/ci/test_cases/index.md index 4bd38f4043a..384bfc10779 100644 --- a/doc/ci/test_cases/index.md +++ b/doc/ci/test_cases/index.md @@ -18,7 +18,9 @@ use external test planning tools, which require additional overhead, context swi ## Create a test case -Users with Reporter or higher [permissions](../../user/permissions.md) can create test cases. +Prerequisite: + +- You must have at least the Reporter [role](../../user/permissions.md). To create a test case in a GitLab project: @@ -32,7 +34,9 @@ To create a test case in a GitLab project: You can view all test cases in the project in the Test Cases list. Filter the issue list with a search query, including labels or the test case's title. -Users with Guest or higher [permissions](../../user/permissions.md) can view test cases. +Prerequisite: + +- You must have at least the Guest [role](../../user/permissions.md). To view a test case: @@ -45,8 +49,10 @@ To view a test case: You can edit a test case's title and description. -Users with Reporter or higher [permissions](../../user/permissions.md) can edit test cases. -Users demoted to the Guest role can continue to edit the test cases they created +Prerequisite: + +- You must have at least the Reporter [role](../../user/permissions.md). +- Users demoted to the Guest role can continue to edit the test cases they created when they were in the higher role. To edit a test case: @@ -60,7 +66,9 @@ To edit a test case: When you want to stop using a test case, you can archive it. You can [reopen an archived test case](#reopen-an-archived-test-case) later. -Users with Reporter or higher [permissions](../../user/permissions.md) can archive test cases. +Prerequisite: + +- You must have at least the Reporter [role](../../user/permissions.md). To archive a test case, on the test case's page, select the **Archive test case** button. @@ -73,6 +81,6 @@ To view archived test cases: If you decide to start using an archived test case again, you can reopen it. -Users with Reporter or higher [permissions](../../user/permissions.md) can reopen test cases. +You must have at least the Reporter [role](../../user/permissions.md). To reopen an archived test case, on the test case's page, select **Reopen test case**. diff --git a/doc/ci/triggers/index.md b/doc/ci/triggers/index.md index a2dd4ac91d5..afcf8ae629a 100644 --- a/doc/ci/triggers/index.md +++ b/doc/ci/triggers/index.md @@ -113,6 +113,9 @@ This means that whenever a new tag is pushed on project A, the job runs and the `stage: deploy` ensures that this job runs only after all jobs with `stage: test` complete successfully. +NOTE: +You [cannot use the API to start `when:manual` trigger jobs](https://gitlab.com/gitlab-org/gitlab/-/issues/284086). + ## Triggering a pipeline from a webhook To trigger a job from a webhook of another project you need to add the following diff --git a/doc/ci/troubleshooting.md b/doc/ci/troubleshooting.md index 994e9294ff6..037e8d3497d 100644 --- a/doc/ci/troubleshooting.md +++ b/doc/ci/troubleshooting.md @@ -16,12 +16,15 @@ This guide also lists common issues and possible solutions. An early source of problems can be incorrect syntax. The pipeline shows a `yaml invalid` badge and does not start running if any syntax or formatting problems are found. -### Edit `gitlab-ci.yml` with the Web IDE +### Edit `gitlab-ci.yml` with the pipeline editor -The [GitLab Web IDE](../user/project/web_ide/index.md) offers advanced authoring tools, -including syntax highlighting for the `.gitlab-ci.yml`, and is the recommended editing -experience (rather than the single file editor). It offers code completion suggestions -that ensure you are only using accepted keywords. +The [pipeline editor](pipeline_editor/index.md) is the recommended editing +experience (rather than the single file editor or the Web IDE). It includes: + +- Code completion suggestions that ensure you are only using accepted keywords. +- Automatic syntax highlighting and validation. +- The [CI/CD configuration visualization](pipeline_editor/index.md#visualize-ci-configuration), + a graphical representation of your `.gitlab-ci.yml` file. If you prefer to use another editor, you can use a schema like [the Schemastore `gitlab-ci` schema](https://json.schemastore.org/gitlab-ci) with your editor of choice. @@ -246,6 +249,39 @@ If the merge train pipeline was canceled before the merge request was merged, wi - Add it to the train again. +### Project `group/project` not found or access denied + +This message is shown if configuration is added with [`include`](yaml/index.md#include) and one of the following: + +- The configuration refers to a project that can't be found. +- The user that is running the pipeline is unable to access any included projects. + +To resolve this, check that: + +- The path of the project is in the format `my-group/my-project` and does not include + any folders in the repository. +- The user running the pipeline is a [member of the projects](../user/project/members/index.md#add-users-to-a-project) + that contain the included files. Users must also have the [permission](../user/permissions.md#job-permissions) + to run CI/CD jobs in the same projects. + +### "The parsed YAML is too big" message + +This message displays when the YAML configuration is too large or nested too deeply. +YAML files with a large number of includes, and thousands of lines overall, are +more likely to hit this memory limit. For example, a YAML file that is 200kb is +likely to hit the default memory limit. + +To reduce the configuration size, you can: + +- Check the length of the expanded CI/CD configuration in the pipeline editor's + [merged YAML](pipeline_editor/index.md#view-expanded-configuration) tab. Look for + duplicated configuration that can be removed or simplified. +- Move long or repeated `script` sections into standalone scripts in the project. +- Use [parent and child pipelines](pipelines/parent_child_pipelines.md) to move some + work to jobs in an independent child pipeline. + +On a self-managed instance, you can [increase the size limits](../administration/instance_limits.md#maximum-size-and-depth-of-cicd-configuration-yaml-files). + ## Pipeline warnings Pipeline configuration warnings are shown when you: diff --git a/doc/ci/unit_test_reports.md b/doc/ci/unit_test_reports.md index c37d7f27970..e758fbc91dd 100644 --- a/doc/ci/unit_test_reports.md +++ b/doc/ci/unit_test_reports.md @@ -290,6 +290,24 @@ javascript: - junit.xml ``` +#### Mocha + +The [JUnit Reporter for Mocha](https://github.com/michaelleeallen/mocha-junit-reporter) NPM package can generate test reports for JavaScript +applications. +In the following `.gitlab-ci.yml` example, the `javascript` job uses Mocha to generate the test reports: + +```yaml +javascript: + stage: test + script: + - mocha --reporter mocha-junit-reporter --reporter-options mochaFile=junit.xml + artifacts: + when: always + reports: + junit: + - junit.xml +``` + ### Flutter / Dart example This example `.gitlab-ci.yml` file uses the [JUnit Report](https://pub.dev/packages/junitreport) package to convert the `flutter test` output into JUnit report XML format: diff --git a/doc/ci/variables/index.md b/doc/ci/variables/index.md index 8650adf0351..a0c8dbd4e4a 100644 --- a/doc/ci/variables/index.md +++ b/doc/ci/variables/index.md @@ -121,7 +121,7 @@ job1: - echo This job does not need any variables ``` -Use the [`value` and `description`](../yaml/index.md#prefill-variables-in-manual-pipelines) +Use the [`value` and `description`](../yaml/index.md#variablesdescription) keywords to define [variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) for [manually-triggered pipelines](../pipelines/index.md#run-a-pipeline-manually). @@ -557,7 +557,7 @@ they can be used in job scripts. 1. Save the `.env` file as an [`artifacts:reports:dotenv`](../yaml/index.md#artifactsreportsdotenv) artifact. 1. Set a job in a later stage to receive the artifact by using the [`dependencies`](../yaml/index.md#dependencies) - or the [`needs`](../yaml/index.md#artifact-downloads-with-needs) keywords. + or the [`needs`](../yaml/index.md#needs) keywords. 1. The later job can then [use the variable in scripts](#use-cicd-variables-in-job-scripts). For example, with the [`dependencies`](../yaml/index.md#dependencies) keyword: @@ -579,7 +579,7 @@ deploy: - build ``` -For example, with the [`needs`](../yaml/index.md#artifact-downloads-with-needs) keyword: +For example, with the [`needs:artifacts`](../yaml/index.md#needsartifacts) keyword: ```yaml build: diff --git a/doc/ci/variables/where_variables_can_be_used.md b/doc/ci/variables/where_variables_can_be_used.md index 9fd1e3eb1a4..e32f0391d2e 100644 --- a/doc/ci/variables/where_variables_can_be_used.md +++ b/doc/ci/variables/where_variables_can_be_used.md @@ -63,14 +63,12 @@ because the expansion is done in GitLab before any runner gets the job. #### Nested variable expansion -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48627) in GitLab 13.10. [Deployed behind the `variable_inside_variable` feature flag](../../user/feature_flags.md), disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/297382) in GitLab 14.3. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/297382) in GitLab 14.4. +- [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/48627) in GitLab 13.10. [Deployed behind the `variable_inside_variable` feature flag](../../user/feature_flags.md), disabled by default. +- [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/297382) in GitLab 14.3. +- [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/297382) in GitLab 14.4. +- Feature flag `variable_inside_variable` removed in GitLab 14.5. -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `variable_inside_variable`. On GitLab.com, this feature is available. - -GitLab expands job variable values recursively before sending them to the runner. For example: +GitLab expands job variable values recursively before sending them to the runner. For example, in the following scenario: ```yaml - BUILD_ROOT_DIR: '${CI_BUILDS_DIR}' @@ -78,10 +76,7 @@ GitLab expands job variable values recursively before sending them to the runner - PACKAGE_PATH: '${OUT_PATH}/pkg' ``` -If nested variable expansion is: - -- **Disabled**: the runner receives `${BUILD_ROOT_DIR}/out/pkg`. This is not a valid path. -- **Enabled**: the runner receives a valid, fully-formed path. For example, if `${CI_BUILDS_DIR}` is `/output`, then `PACKAGE_PATH` would be `/output/out/pkg`. +The runner receives a valid, fully-formed path. For example, if `${CI_BUILDS_DIR}` is `/output`, then `PACKAGE_PATH` would be `/output/out/pkg`. References to unavailable variables are left intact. In this case, the runner [attempts to expand the variable value](#gitlab-runner-internal-variable-expansion-mechanism) at runtime. diff --git a/doc/ci/yaml/gitlab_ci_yaml.md b/doc/ci/yaml/gitlab_ci_yaml.md index ea05aa45b0b..9bab4a7fc77 100644 --- a/doc/ci/yaml/gitlab_ci_yaml.md +++ b/doc/ci/yaml/gitlab_ci_yaml.md @@ -24,7 +24,7 @@ In the `.gitlab-ci.yml` file, you can define: The scripts are grouped into **jobs**, and jobs run as part of a larger **pipeline**. You can group multiple independent jobs into **stages** that run in a defined order. -The CI/CD configuration needs at least one job that is [not hidden](index.md#hide-jobs). +The CI/CD configuration needs at least one job that is [not hidden](../jobs/index.md#hide-jobs). You should organize your jobs in a sequence that suits your application and is in accordance with the tests you wish to perform. To [visualize](../pipeline_editor/index.md#visualize-ci-configuration) the process, imagine diff --git a/doc/ci/yaml/includes.md b/doc/ci/yaml/includes.md index 92bf44cca7f..5e2eb53a0ea 100644 --- a/doc/ci/yaml/includes.md +++ b/doc/ci/yaml/includes.md @@ -69,7 +69,7 @@ You can include an array of configuration files: ## Use `default` configuration from an included configuration file -You can define a [`default`](index.md#custom-default-keyword-values) section in a +You can define a [`default`](index.md#default) section in a configuration file. When you use a `default` section with the `include` keyword, the defaults apply to all jobs in the pipeline. @@ -216,3 +216,120 @@ default: after_script: - echo "Job complete." ``` + +## Use variables with `include` + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/284883) in GitLab 13.8. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/294294) in GitLab 13.9. +> - [Support for project, group, and instance variables added](https://gitlab.com/gitlab-org/gitlab/-/issues/219065) in GitLab 14.2. +> - [Support for pipeline variables added](https://gitlab.com/gitlab-org/gitlab/-/issues/337633) in GitLab 14.5. + +In `include` sections in your `.gitlab-ci.yml` file, you can use: + +- [Project variables](../variables/index.md#add-a-cicd-variable-to-a-project) +- [Group variables](../variables/index.md#add-a-cicd-variable-to-a-group) +- [Instance variables](../variables/index.md#add-a-cicd-variable-to-an-instance) +- Project [predefined variables](../variables/predefined_variables.md) +- In GitLab 14.2 and later, the `$CI_COMMIT_REF_NAME` [predefined variable](../variables/predefined_variables.md). + + When used in `include`, the `CI_COMMIT_REF_NAME` variable returns the full + ref path, like `refs/heads/branch-name`. In `include:rules`, you might need to use + `if: $CI_COMMIT_REF_NAME =~ /main/` (not `== main`). This behavior is resolved in GitLab 14.5. + +In GitLab 14.5 and later, you can also use: + +- [Trigger variables](../triggers/index.md#making-use-of-trigger-variables). +- [Scheduled pipeline variables](../pipelines/schedules.md#using-variables). +- [Manual pipeline run variables](../variables/index.md#override-a-variable-when-running-a-pipeline-manually). +- Pipeline [predefined variables](../variables/predefined_variables.md). + + YAML files are parsed before the pipeline is created, so the following pipeline predefined variables + are **not** available: + + - `CI_PIPELINE_ID` + - `CI_PIPELINE_URL` + - `CI_PIPELINE_IID` + - `CI_PIPELINE_CREATED_AT` + +For example: + +```yaml +include: + project: '$CI_PROJECT_PATH' + file: '.compliance-gitlab-ci.yml' +``` + +For an example of how you can include these predefined variables, and the variables' impact on CI/CD jobs, +see this [CI/CD variable demo](https://youtu.be/4XR8gw3Pkos). + +## Use `rules` with `include` + +> - Introduced in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) GitLab 14.3. +> - [Feature flag `ci_include_rules` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.4. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.4. +> - [Support for `exists` keyword added](https://gitlab.com/gitlab-org/gitlab/-/issues/341511) in GitLab 14.5. + +You can use [`rules`](index.md#rules) with `include` to conditionally include other configuration files. + +You can only use the following rules with `include` (and only with [certain variables](#use-variables-with-include)): + +- [`if` rules](index.md#rulesif). For example: + + ```yaml + include: + - local: builds.yml + rules: + - if: '$INCLUDE_BUILDS == "true"' + - local: deploys.yml + rules: + - if: $CI_COMMIT_BRANCH == "main" + + test: + stage: test + script: exit 0 + ``` + +- [`exists` rules](index.md#rulesexists). For example: + + ```yaml + include: + - local: builds.yml + rules: + - exists: + - file.md + + test: + stage: test + script: exit 0 + ``` + +`rules` keyword `changes` is not supported. + +## Use `include:local` with wildcard file paths + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25921) in GitLab 13.11. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/327315) in GitLab 14.2. + +You can use wildcard paths (`*` and `**`) with `include:local`. + +Example: + +```yaml +include: 'configs/*.yml' +``` + +When the pipeline runs, GitLab: + +- Adds all `.yml` files in the `configs` directory into the pipeline configuration. +- Does not add `.yml` files in subfolders of the `configs` directory. To allow this, + add the following configuration: + + ```yaml + # This matches all `.yml` files in `configs` and any subfolder in it. + include: 'configs/**.yml' + + # This matches all `.yml` files only in subfolders of `configs`. + include: 'configs/**/*.yml' + ``` diff --git a/doc/ci/yaml/index.md b/doc/ci/yaml/index.md index aa44400fffc..5702c7a7dfd 100644 --- a/doc/ci/yaml/index.md +++ b/doc/ci/yaml/index.md @@ -16,67 +16,68 @@ This document lists the configuration options for your GitLab `.gitlab-ci.yml` f When you are editing your `.gitlab-ci.yml` file, you can validate it with the [CI Lint](../lint.md) tool. -## Job keywords +## Keywords + +A GitLab CI/CD pipeline configuration includes: + +- [Global keywords](#global-keywords) that configure pipeline behavior: + + | Keyword | Description | + |-------------------------|:------------| + | [`default`](#default) | Custom default values for job keywords. | + | [`stages`](#stages) | The names and order of the pipeline stages. | + | [`workflow`](#workflow) | Control what types of pipeline run. | + | [`include`](#include) | Import configuration from other YAML files. | + +- [Jobs](../jobs/index.md) configured with [job keywords](#job-keywords): + + | Keyword | Description | + | :-------------------------------------------|:------------| + | [`after_script`](#after_script) | Override a set of commands that are executed after job. | + | [`allow_failure`](#allow_failure) | Allow job to fail. A failed job does not cause the pipeline to fail. | + | [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. | + | [`before_script`](#before_script) | Override a set of commands that are executed before job. | + | [`cache`](#cache) | List of files that should be cached between subsequent runs. | + | [`coverage`](#coverage) | Code coverage settings for a given job. | + | [`dast_configuration`](#dast_configuration) | Use configuration from DAST profiles on a job level. | + | [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | + | [`environment`](#environment) | Name of an environment to which the job deploys. | + | [`except`](#only--except) | Control when jobs are not created. | + | [`extends`](#extends) | Configuration entries that this job inherits from. | + | [`image`](#image) | Use Docker images. | + | [`inherit`](#inherit) | Select which global defaults all jobs inherit. | + | [`interruptible`](#interruptible) | Defines if a job can be canceled when made redundant by a newer run. | + | [`needs`](#needs) | Execute jobs earlier than the stage ordering. | + | [`only`](#only--except) | Control when jobs are created. | + | [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. | + | [`parallel`](#parallel) | How many instances of a job should be run in parallel. | + | [`release`](#release) | Instructs the runner to generate a [release](../../user/project/releases/index.md) object. | + | [`resource_group`](#resource_group) | Limit job concurrency. | + | [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | + | [`rules`](#rules) | List of conditions to evaluate and determine selected attributes of a job, and whether or not it's created. | + | [`script`](#script) | Shell script that is executed by a runner. | + | [`secrets`](#secrets) | The CI/CD secrets the job needs. | + | [`services`](#services) | Use Docker services images. | + | [`stage`](#stage) | Defines a job stage. | + | [`tags`](#tags) | List of tags that are used to select a runner. | + | [`timeout`](#timeout) | Define a custom job-level timeout that takes precedence over the project-wide setting. | + | [`trigger`](#trigger) | Defines a downstream pipeline trigger. | + | [`variables`](#variables) | Define job variables on a job level. | + | [`when`](#when) | When to run job. | -A job is defined as a list of keywords that define the job's behavior. - -The keywords available for jobs are: - -| Keyword | Description | -| :-------------------------------------------|:------------| -| [`after_script`](#after_script) | Override a set of commands that are executed after job. | -| [`allow_failure`](#allow_failure) | Allow job to fail. A failed job does not cause the pipeline to fail. | -| [`artifacts`](#artifacts) | List of files and directories to attach to a job on success. | -| [`before_script`](#before_script) | Override a set of commands that are executed before job. | -| [`cache`](#cache) | List of files that should be cached between subsequent runs. | -| [`coverage`](#coverage) | Code coverage settings for a given job. | -| [`dast_configuration`](#dast_configuration) | Use configuration from DAST profiles on a job level. | -| [`dependencies`](#dependencies) | Restrict which artifacts are passed to a specific job by providing a list of jobs to fetch artifacts from. | -| [`environment`](#environment) | Name of an environment to which the job deploys. | -| [`except`](#only--except) | Control when jobs are not created. | -| [`extends`](#extends) | Configuration entries that this job inherits from. | -| [`image`](#image) | Use Docker images. | -| [`include`](#include) | Include external YAML files. | -| [`inherit`](#inherit) | Select which global defaults all jobs inherit. | -| [`interruptible`](#interruptible) | Defines if a job can be canceled when made redundant by a newer run. | -| [`needs`](#needs) | Execute jobs earlier than the stage ordering. | -| [`only`](#only--except) | Control when jobs are created. | -| [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. | -| [`parallel`](#parallel) | How many instances of a job should be run in parallel. | -| [`release`](#release) | Instructs the runner to generate a [release](../../user/project/releases/index.md) object. | -| [`resource_group`](#resource_group) | Limit job concurrency. | -| [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. | -| [`rules`](#rules) | List of conditions to evaluate and determine selected attributes of a job, and whether or not it's created. | -| [`script`](#script) | Shell script that is executed by a runner. | -| [`secrets`](#secrets) | The CI/CD secrets the job needs. | -| [`services`](#services) | Use Docker services images. | -| [`stage`](#stage) | Defines a job stage. | -| [`tags`](#tags) | List of tags that are used to select a runner. | -| [`timeout`](#timeout) | Define a custom job-level timeout that takes precedence over the project-wide setting. | -| [`trigger`](#trigger) | Defines a downstream pipeline trigger. | -| [`variables`](#variables) | Define job variables on a job level. | -| [`when`](#when) | When to run job. | - -### Unavailable names for jobs - -You can't use these keywords as job names: - -- `image` -- `services` -- `stages` -- `types` -- `before_script` -- `after_script` -- `variables` -- `cache` -- `include` +## Global keywords + +Some keywords are not defined in a job. These keywords control pipeline behavior +or import additional pipeline configuration. -### Custom default keyword values +### `default` You can set global defaults for some keywords. Jobs that do not define one or more of the listed keywords use the value defined in the `default:` section. -These job keywords can be defined inside a `default:` section: +**Keyword type**: Global keyword. + +**Possible inputs**: These keywords can have custom defaults: - [`after_script`](#after_script) - [`artifacts`](#artifacts) @@ -89,9 +90,7 @@ These job keywords can be defined inside a `default:` section: - [`tags`](#tags) - [`timeout`](#timeout) -The following example sets the `ruby:3.0` image as the default for all jobs in the pipeline. -The `rspec 2.7` job does not use the default, because it overrides the default with -a job-specific `image:` section: +**Example of `default`:** ```yaml default: @@ -105,25 +104,24 @@ rspec 2.7: script: bundle exec rspec ``` -## Global keywords +In this example, `ruby:3.0` is the default `image` value for all jobs in the pipeline. +The `rspec 2.7` job does not use the default, because it overrides the default with +a job-specific `image:` section: -Some keywords are not defined in a job. These keywords control pipeline behavior -or import additional pipeline configuration: +**Additional details**: -| Keyword | Description | -|-------------------------|:------------| -| [`stages`](#stages) | The names and order of the pipeline stages. | -| [`workflow`](#workflow) | Control what types of pipeline run. | -| [`include`](#include) | Import configuration from other YAML files. | +- When the pipeline is created, each default is copied to all jobs that don't have + that keyword defined. +- If a job already has one of the keywords configured, the configuration in the job + takes precedence and is not replaced by the default. +- Control inheritance of default keywords in jobs with [`inherit:default`](#inheritdefault). ### `stages` -Use `stages` to define stages that contain groups of jobs. `stages` is defined globally -for the pipeline. Use [`stage`](#stage) in a job to define which stage the job is -part of. +Use `stages` to define stages that contain groups of jobs. Use [`stage`](#stage) +in a job to configure the job to run in a specific stage. -If `stages` is not defined in the `.gitlab-ci.yml` file, then the default -pipeline stages are: +If `stages` is not defined in the `.gitlab-ci.yml` file, the default pipeline stages are: - [`.pre`](#stage-pre) - `build` @@ -131,12 +129,14 @@ pipeline stages are: - `deploy` - [`.post`](#stage-post) -The order of the `stages` items defines the execution order for jobs: +The order of the items in `stages` defines the execution order for jobs: - Jobs in the same stage run in parallel. - Jobs in the next stage run after the jobs from the previous stage complete successfully. -For example: +**Keyword type**: Global keyword. + +**Example of `stages`:** ```yaml stages: @@ -145,6 +145,8 @@ stages: - deploy ``` +In this example: + 1. All jobs in `build` execute in parallel. 1. If all jobs in `build` succeed, the `test` jobs execute in parallel. 1. If all jobs in `test` succeed, the `deploy` jobs execute in parallel. @@ -153,94 +155,92 @@ stages: If any job fails, the pipeline is marked as `failed` and jobs in later stages do not start. Jobs in the current stage are not stopped and continue to run. -If a job does not specify a [`stage`](#stage), the job is assigned the `test` stage. +**Additional details**: -If a stage is defined, but no jobs use it, the stage is not visible in the pipeline. This is -useful for [compliance pipeline configuration](../../user/project/settings/index.md#compliance-pipeline-configuration) -because: +- If a job does not specify a [`stage`](#stage), the job is assigned the `test` stage. +- If a stage is defined but no jobs use it, the stage is not visible in the pipeline, + which can help [compliance pipeline configurations](../../user/project/settings/index.md#compliance-pipeline-configuration): + - Stages can be defined in the compliance configuration but remain hidden if not used. + - The defined stages become visible when developers use them in job definitions. -- Stages can be defined in the compliance configuration but remain hidden if not used. -- The defined stages become visible when developers use them in job definitions. +**Related topics**: -To make a job start earlier and ignore the stage order, use -the [`needs`](#needs) keyword. +- To make a job start earlier and ignore the stage order, use the [`needs`](#needs) keyword. ### `workflow` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/29654) in GitLab 12.5 -Use `workflow:` to determine whether or not a pipeline is created. -Define this keyword at the top level, with a single `rules:` keyword that -is similar to [`rules:` defined in jobs](#rules). +Use [`workflow`](workflow.md) to control pipeline behavior. -You can use the [`workflow:rules` templates](#workflowrules-templates) to import -a preconfigured `workflow: rules` entry. +**Related topics**: -`workflow: rules` accepts these keywords: +- [`workflow: rules` examples](workflow.md#workflow-rules-examples) +- [Switch between branch pipelines and merge request pipelines](workflow.md#switch-between-branch-pipelines-and-merge-request-pipelines) -- [`if`](#rulesif): Check this rule to determine when to run a pipeline. -- [`when`](#when): Specify what to do when the `if` rule evaluates to true. - - To run a pipeline, set to `always`. - - To prevent pipelines from running, set to `never`. -- [`variables`](#workflowrulesvariables): If not defined, uses the [variables defined elsewhere](#variables). +#### `workflow:rules` -When no rules evaluate to true, the pipeline does not run. +The `rules` keyword in `workflow` is similar to [`rules:` defined in jobs](#rules), +but controls whether or not a whole pipeline is created. -Some example `if` clauses for `workflow: rules`: +When no rules evaluate to true, the pipeline does not run. -| Example rules | Details | -|------------------------------------------------------|-----------------------------------------------------------| -| `if: '$CI_PIPELINE_SOURCE == "merge_request_event"'` | Control when merge request pipelines run. | -| `if: '$CI_PIPELINE_SOURCE == "push"'` | Control when both branch pipelines and tag pipelines run. | -| `if: $CI_COMMIT_TAG` | Control when tag pipelines run. | -| `if: $CI_COMMIT_BRANCH` | Control when branch pipelines run. | +**Possible inputs**: You can use some of the same keywords as job-level [`rules`](#rules): -See the [common `if` clauses for `rules`](../jobs/job_control.md#common-if-clauses-for-rules) for more examples. +- [`rules: if`](#rulesif). +- [`rules: changes`](#ruleschanges). +- [`rules: exists`](#rulesexists). +- [`when`](#when), can only be `always` or `never` when used with `workflow`. +- [`variables`](#workflowrulesvariables). -In the following example, pipelines run for all `push` events (changes to -branches and new tags). Pipelines for push events with `-draft` in the commit message -don't run, because they are set to `when: never`. Pipelines for schedules or merge requests -don't run either, because no rules evaluate to true for them: +**Example of `workflow:rules`:** ```yaml workflow: rules: - if: $CI_COMMIT_MESSAGE =~ /-draft$/ when: never - - if: '$CI_PIPELINE_SOURCE == "push"' + - if: $CI_PIPELINE_SOURCE == "merge_request_event" + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH ``` -This example has strict rules, and pipelines do **not** run in any other case. +In this example, pipelines run if the commit message does not have `-drafts` in it +and the pipeline is for either: -Alternatively, all of the rules can be `when: never`, with a final -`when: always` rule. Pipelines that match the `when: never` rules do not run. -All other pipeline types run: +- A merge request +- The default branch. -```yaml -workflow: - rules: - - if: '$CI_PIPELINE_SOURCE == "schedule"' - when: never - - if: '$CI_PIPELINE_SOURCE == "push"' - when: never - - when: always -``` +**Additional details**: -This example prevents pipelines for schedules or `push` (branches and tags) pipelines. -The final `when: always` rule runs all other pipeline types, **including** merge -request pipelines. +- If your rules match both branch pipelines (other than the default branch) and merge request pipelines, + [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) can occur. -If your rules match both branch pipelines and merge request pipelines, -[duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) can occur. +**Related topics**: + +- You can use the [`workflow:rules` templates](workflow.md#workflowrules-templates) to import + a preconfigured `workflow: rules` entry. +- [Common `if` clauses for `workflow:rules`](workflow.md#common-if-clauses-for-workflowrules). #### `workflow:rules:variables` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294232) in GitLab 13.11. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/300997) in GitLab 14.1. -You can use [`variables`](#variables) in `workflow:rules:` to define variables for specific pipeline conditions. +You can use [`variables`](#variables) in `workflow:rules:` to define variables for +specific pipeline conditions. -For example: +When the condition matches, the variable is created and can be used by all jobs +in the pipeline. If the variable is already defined at the global level, the `workflow` +variable takes precedence and overrides the global variable. + +**Keyword type**: Global keyword. + +**Possible inputs**: Variable name and value pairs: + +- The name can use only numbers, letters, and underscores (`_`). +- The value must be a string. + +**Example of `workflow:rules:variables`:** ```yaml variables: @@ -289,198 +289,61 @@ When the branch is something else: - job1's `DEPLOY_VARIABLE` is `job1-default-deploy`. - job2's `DEPLOY_VARIABLE` is `default-deploy`. -#### `workflow:rules` templates - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217732) in GitLab 13.0. - -GitLab provides templates that set up `workflow: rules` -for common scenarios. These templates help prevent duplicate pipelines. - -The [`Branch-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/Branch-Pipelines.gitlab-ci.yml) -makes your pipelines run for branches and tags. - -Branch pipeline status is displayed in merge requests that use the branch -as a source. However, this pipeline type does not support any features offered by -[merge request pipelines](../pipelines/merge_request_pipelines.md), like -[pipelines for merged results](../pipelines/pipelines_for_merged_results.md) -or [merge trains](../pipelines/merge_trains.md). -This template intentionally avoids those features. - -To [include](#include) it: - -```yaml -include: - - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' -``` - -The [`MergeRequest-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/MergeRequest-Pipelines.gitlab-ci.yml) -makes your pipelines run for the default branch, tags, and -all types of merge request pipelines. Use this template if you use any of the -the [pipelines for merge requests features](../pipelines/merge_request_pipelines.md). - -To [include](#include) it: - -```yaml -include: - - template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml' -``` - -#### Switch between branch pipelines and merge request pipelines - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) in GitLab 13.8. - -To make the pipeline switch from branch pipelines to merge request pipelines after -a merge request is created, add a `workflow: rules` section to your `.gitlab-ci.yml` file. - -If you use both pipeline types at the same time, [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) -might run at the same time. To prevent duplicate pipelines, use the -[`CI_OPEN_MERGE_REQUESTS` variable](../variables/predefined_variables.md). - -The following example is for a project that runs branch and merge request pipelines only, -but does not run pipelines for any other case. It runs: - -- Branch pipelines when a merge request is not open for the branch. -- Merge request pipelines when a merge request is open for the branch. - -```yaml -workflow: - rules: - - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' - - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS' - when: never - - if: '$CI_COMMIT_BRANCH' -``` - -If the pipeline is triggered by: - -- A merge request, run a merge request pipeline. For example, a merge request pipeline - can be triggered by a push to a branch with an associated open merge request. -- A change to a branch, but a merge request is open for that branch, do not run a branch pipeline. -- A change to a branch, but without any open merge requests, run a branch pipeline. - -You can also add a rule to an existing `workflow` section to switch from branch pipelines -to merge request pipelines when a merge request is created. - -Add this rule to the top of the `workflow` section, followed by the other rules that -were already present: - -```yaml -workflow: - rules: - - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" - when: never - - ... # Previously defined workflow rules here -``` - -[Triggered pipelines](../triggers/index.md) that run on a branch have a `$CI_COMMIT_BRANCH` -set and could be blocked by a similar rule. Triggered pipelines have a pipeline source -of `trigger` or `pipeline`, so `&& $CI_PIPELINE_SOURCE == "push"` ensures the rule -does not block triggered pipelines. - ### `include` > [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/42861) to GitLab Free in 11.4. Use `include` to include external YAML files in your CI/CD configuration. -You can break down one long `.gitlab-ci.yml` file into multiple files to increase readability, +You can split one long `.gitlab-ci.yml` file into multiple files to increase readability, or reduce duplication of the same configuration in multiple places. -You can also store template files in a central repository and `include` them in projects. - -`include` requires the external YAML file to have the extensions `.yml` or `.yaml`, -otherwise the external file is not included. - -You can't use [YAML anchors](#anchors) across different YAML files sourced by `include`. -You can only refer to anchors in the same file. To reuse configuration from different -YAML files, use [`!reference` tags](#reference-tags) or the [`extends` keyword](#extends). - -`include` supports the following inclusion methods: - -| Keyword | Method | -|:--------------------------------|:------------------------------------------------------------------| -| [`local`](#includelocal) | Include a file from the local project repository. | -| [`file`](#includefile) | Include a file from a different project repository. | -| [`remote`](#includeremote) | Include a file from a remote URL. Must be publicly accessible. | -| [`template`](#includetemplate) | Include templates that are provided by GitLab. | - -When the pipeline starts, the `.gitlab-ci.yml` file configuration included by all methods is evaluated. -The configuration is a snapshot in time and persists in the database. GitLab does not reflect any changes to -the referenced `.gitlab-ci.yml` file configuration until the next pipeline starts. +You can also store template files in a central repository and include them in projects. The `include` files are: -- Deep merged with those in the `.gitlab-ci.yml` file. -- Always evaluated first and merged with the content of the `.gitlab-ci.yml` file, +- Merged with those in the `.gitlab-ci.yml` file. +- Always evaluated first and then merged with the content of the `.gitlab-ci.yml` file, regardless of the position of the `include` keyword. -NOTE: -Use merging to customize and override included CI/CD configurations with local -configurations. Local configurations in the `.gitlab-ci.yml` file override included configurations. - -#### Variables with `include` - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/284883) in GitLab 13.8. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/294294) in GitLab 13.9. -> - [Support for project, group, and instance variables added](https://gitlab.com/gitlab-org/gitlab/-/issues/219065) in GitLab 14.2. +You can [nest](includes.md#use-nested-includes) up to 100 includes, but you can't have duplicate includes. +In [GitLab 12.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/28212), +the time limit to resolve all files is 30 seconds. -In `include` sections in your `.gitlab-ci.yml` file, you can use: +**Keyword type**: Global keyword. -- `$CI_COMMIT_REF_NAME` [predefined variable](../variables/predefined_variables.md) in GitLab 14.2 - and later. -- [Project variables](../variables/index.md#add-a-cicd-variable-to-a-project) -- [Group variables](../variables/index.md#add-a-cicd-variable-to-a-group) -- [Instance variables](../variables/index.md#add-a-cicd-variable-to-an-instance) -- Project [predefined variables](../variables/predefined_variables.md). +**Possible inputs**: The `include` subkeys: -```yaml -include: - project: '$CI_PROJECT_PATH' - file: '.compliance-gitlab-ci.yml' -``` - -For an example of how you can include these predefined variables, and the variables' impact on CI/CD jobs, -see this [CI/CD variable demo](https://youtu.be/4XR8gw3Pkos). - -There is a [related issue](https://gitlab.com/gitlab-org/gitlab/-/issues/337633) -that proposes expanding this feature to support more variables. +- [`include:local`](#includelocal) +- [`include:file`](#includefile) +- [`include:remote`](#includeremote) +- [`include:template`](#includetemplate) -#### `rules` with `include` - -> - Introduced in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) GitLab 14.3. -> - [Feature flag `ci_include_rules` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.4. -> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.4. +**Additional details**: -You can use [`rules`](#rules) with `include` to conditionally include other configuration files. -You can only use [`if` rules](#rulesif) in `include`, and only with [certain variables](#variables-with-include). -`rules` keywords such as `changes` and `exists` are not supported. +- Use merging to customize and override included CI/CD configurations with local +- You can override included configuration by having the same job name or global keyword + in the `.gitlab-ci.yml` file. The two configurations are merged together, and the + configuration in the `.gitlab-ci.yml` file takes precedence over the included configuration. -```yaml -include: - - local: builds.yml - rules: - - if: '$INCLUDE_BUILDS == "true"' +**Related topics**: -test: - stage: test - script: exit 0 -``` +- [Use variables with `include`](includes.md#use-variables-with-include). +- [Use `rules` with `include`](includes.md#use-rules-with-include). #### `include:local` Use `include:local` to include a file that is in the same repository as the `.gitlab-ci.yml` file. -Use a full path relative to the root directory (`/`). +Use `include:local` instead of symbolic links. -If you use `include:local`, make sure that both the `.gitlab-ci.yml` file and the local file -are on the same branch. +**Keyword type**: Global keyword. -You can't include local files through Git submodules paths. +**Possible inputs**: -All [nested includes](#nested-includes) are executed in the scope of the same project, -so it's possible to use local, project, remote, or template includes. +- A full path relative to the root directory (`/`). +- The YAML file must have the extension `.yml` or `.yaml`. +- You can [use `*` and `**` wildcards in the file path](includes.md#use-includelocal-with-wildcard-file-paths). -Example: +**Example of `include:local`**: ```yaml include: @@ -493,44 +356,26 @@ You can also use shorter syntax to define the path: include: '.gitlab-ci-production.yml' ``` -Use local includes instead of symbolic links. - -##### `include:local` with wildcard file paths - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25921) in GitLab 13.11. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/327315) in GitLab 14.2. - -You can use wildcard paths (`*` and `**`) with `include:local`. - -Example: - -```yaml -include: 'configs/*.yml' -``` - -When the pipeline runs, GitLab: - -- Adds all `.yml` files in the `configs` directory into the pipeline configuration. -- Does not add `.yml` files in subfolders of the `configs` directory. To allow this, - add the following configuration: - - ```yaml - # This matches all `.yml` files in `configs` and any subfolder in it. - include: 'configs/**.yml' +**Additional details**: - # This matches all `.yml` files only in subfolders of `configs`. - include: 'configs/**/*.yml' - ``` +- The `.gitlab-ci.yml` file and the local file must be on the same branch. +- You can't include local files through Git submodules paths. +- All [nested includes](includes.md#use-nested-includes) are executed in the scope of the same project, + so you can use local, project, remote, or template includes. #### `include:file` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53903) in GitLab 11.7. +> Including multiple files from the same project [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26793) in GitLab 13.6. [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/271560) in GitLab 13.8. To include files from another private project on the same GitLab instance, use `include:file`. You can use `include:file` in combination with `include:project` only. -Use a full path, relative to the root directory (`/`). -For example: +**Keyword type**: Global keyword. + +**Possible inputs**: A full path, relative to the root directory (`/`). +The YAML file must have the extension `.yml` or `.yaml`. + +**Example of `include:file`**: ```yaml include: @@ -555,14 +400,6 @@ include: file: '/templates/.gitlab-ci-template.yml' ``` -All [nested includes](#nested-includes) are executed in the scope of the target project. -You can use local (relative to target project), project, remote, or template includes. - -##### Multiple files from a project - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26793) in GitLab 13.6. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/271560) in GitLab 13.8. - You can include multiple files from the same project: ```yaml @@ -574,28 +411,52 @@ include: - '/templates/.tests.yml' ``` +**Additional details**: + +- All [nested includes](includes.md#use-nested-includes) are executed in the scope of the target project. + You can use `local` (relative to the target project), `project`, `remote`, or `template` includes. +- When the pipeline starts, the `.gitlab-ci.yml` file configuration included by all methods is evaluated. + The configuration is a snapshot in time and persists in the database. GitLab does not reflect any changes to + the referenced `.gitlab-ci.yml` file configuration until the next pipeline starts. +- When you include a YAML file from another private project, the user running the pipeline + must be a member of both projects and have the appropriate permissions to run pipelines. + A `not found or access denied` error may be displayed if the user does not have access to any of the included files. + #### `include:remote` Use `include:remote` with a full URL to include a file from a different location. -The remote file must be publicly accessible by an HTTP/HTTPS `GET` request, because -authentication in the remote URL is not supported. For example: + +**Keyword type**: Global keyword. + +**Possible inputs**: A public URL accessible by an HTTP/HTTPS `GET` request. +Authentication with the remote URL is not supported. + +The YAML file must have the extension `.yml` or `.yaml`. + +**Example of `include:remote`**: ```yaml include: - remote: 'https://gitlab.com/example-project/-/raw/main/.gitlab-ci.yml' ``` -All [nested includes](#nested-includes) execute without context as a public user, -so you can only `include` public projects or templates. +**Additional details**: + +- All [nested includes](includes.md#use-nested-includes) execute without context as a public user, + so you can only include public projects or templates. +- Be careful when including a remote CI/CD configuration file. No pipelines or notifications + trigger when external CI/CD configuration files change. From a security perspective, + this is similar to pulling a third-party dependency. #### `include:template` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53445) in GitLab 11.7. +Use `include:template` to include [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). -Use `include:template` to include `.gitlab-ci.yml` templates that are -[shipped with GitLab](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). +**Keyword type**: Global keyword. -For example: +**Possible inputs**: [`.gitlab-ci.yml` templates](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates). + +**Example of `include:template`**: ```yaml # File sourced from the GitLab template collection @@ -611,82 +472,145 @@ include: - template: Auto-DevOps.gitlab-ci.yml ``` -All [nested includes](#nested-includes) are executed only with the permission of the user, -so it's possible to use project, remote or template includes. +**Additional details**: -#### Nested includes +- All [nested includes](includes.md#use-nested-includes) are executed only with the permission of the user, + so it's possible to use `project`, `remote`, or `template` includes. -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/56836) in GitLab 11.9. +## Job keywords -Use nested includes to compose a set of includes. +The following topics explain how to use keywords to configure CI/CD pipelines. -You can have up to 100 includes, but you can't have duplicate includes. +### `image` -In [GitLab 12.4 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/28212), the time limit -to resolve all files is 30 seconds. +Use `image` to specify a Docker image that the job runs in. -#### Additional `includes` examples +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). -View [additional `includes` examples](includes.md). +**Possible inputs**: The name of the image, including the registry path if needed, in one of these formats: -## Keyword details +- `<image-name>` (Same as using `<image-name>` with the `latest` tag) +- `<image-name>:<tag>` +- `<image-name>@<digest>` -The following topics explain how to use keywords to configure CI/CD pipelines. +**Example of `image`**: -### `image` +```yaml +default: + image: ruby:3.0 -Use `image` to specify [a Docker image](../docker/using_docker_images.md#what-is-an-image) to use for the job. +rspec: + script: bundle exec rspec -For: +rspec 2.7: + image: registry.example.com/my-group/my-project/ruby:2.7 + script: bundle exec rspec +``` -- Usage examples, see [Define `image` in the `.gitlab-ci.yml` file](../docker/using_docker_images.md#define-image-in-the-gitlab-ciyml-file). -- Detailed usage information, refer to [Docker integration](../docker/index.md) documentation. +In this example, the `ruby:3.0` image is the default for all jobs in the pipeline. +The `rspec 2.7` job does not use the default, because it overrides the default with +a job-specific `image:` section. + +**Related topics**: + +- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). #### `image:name` -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +The name of the Docker image that the job runs in. Similar to [`image:`](#image) used by itself. + +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). + +**Possible inputs**: The name of the image, including the registry path if needed, in one of these formats: + +- `<image-name>` (Same as using `<image-name>` with the `latest` tag) +- `<image-name>:<tag>` +- `<image-name>@<digest>` -For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). +**Example of `image:name`**: + +```yaml +image: + name: "registry.example.com/my/image:latest" +``` + +**Related topics**: + +- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). #### `image:entrypoint` -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +Command or script to execute as the container's entry point. -For more information, see [Available settings for `image`](../docker/using_docker_images.md#available-settings-for-image). +When the Docker container is created, the `entrypoint` is translated to the Docker `--entrypoint` option. +The syntax is similar to the [Dockerfile `ENTRYPOINT` directive](https://docs.docker.com/engine/reference/builder/#entrypoint), +where each shell token is a separate string in the array. -#### `services` +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). -Use `services` to specify a [service Docker image](../services/index.md), linked to a base image specified in [`image`](#image). +**Possible inputs**: A string. -For: +**Example of `image:entrypoint`**: -- Usage examples, see [Define `services` in the `.gitlab-ci.yml` file](../services/index.md#define-services-in-the-gitlab-ciyml-file). -- Detailed usage information, refer to [Docker integration](../docker/index.md) documentation. -- Example services, see [GitLab CI/CD Services](../services/index.md). +```yaml +image: + name: super/sql:experimental + entrypoint: [""] +``` + +**Related topics**: + +- [Override the entrypoint of an image](../docker/using_docker_images.md#override-the-entrypoint-of-an-image). -##### `services:name` +#### `services` + +Use `services` to specify an additional Docker image to run scripts in. The [`services` image](../services/index.md) is linked +to the image specified in the [`image`](#image) keyword. -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). -For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). +**Possible inputs**: The name of the services image, including the registry path if needed, in one of these formats: -##### `services:alias` +- `<image-name>` (Same as using `<image-name>` with the `latest` tag) +- `<image-name>:<tag>` +- `<image-name>@<digest>` -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +**Example of `services`**: -For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). +```yaml +default: + image: + name: ruby:2.6 + entrypoint: ["/bin/bash"] -##### `services:entrypoint` + services: + - name: my-postgres:11.7 + alias: db-postgres + entrypoint: ["/usr/local/bin/db-postgres"] + command: ["start"] -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). + before_script: + - bundle install -For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). +test: + script: + - bundle exec rake spec +``` -##### `services:command` +In this example, the job launches a Ruby container. Then, from that container, the job launches +another container that's running PostgreSQL. Then the job then runs scripts +in that container. -An [extended Docker configuration option](../docker/using_docker_images.md#extended-docker-configuration-options). +**Related topics**: -For more information, see [Available settings for `services`](../services/index.md#available-settings-for-services). +- [Available settings for `services`](../services/index.md#available-settings-for-services). +- [Define `services` in the `.gitlab-ci.yml` file](../services/index.md#define-services-in-the-gitlab-ciyml-file). +- [Run your CI/CD jobs in Docker containers](../docker/using_docker_images.md). +- [Use Docker to build Docker images](../docker/using_docker_build.md). ### `script` @@ -700,7 +624,7 @@ All jobs except [trigger jobs](#trigger) require a `script` keyword. - Single line commands. - Long commands [split over multiple lines](script.md#split-long-commands). -- [YAML anchors](#yaml-anchors-for-scripts). +- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). **Example of `script`:** @@ -716,8 +640,7 @@ job2: **Additional details**: -You might need to use single quotes (`'`) or double quotes (`"`) when using -[special characters in `script`](script.md#use-special-characters-with-script). +- When you use [these special characters in `script`](script.md#use-special-characters-with-script), you must use single quotes (`'`) or double quotes (`"`) . **Related topics**: @@ -733,13 +656,13 @@ Use `before_script` to define an array of commands that should run before each j `script` commands, but after [artifacts](#artifacts) are restored. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#custom-default-keyword-values). +[`default:` section](#default). **Possible inputs**: An array including: - Single line commands. - Long commands [split over multiple lines](script.md#split-long-commands). -- [YAML anchors](#yaml-anchors-for-scripts). +- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). **Example of `before_script`:** @@ -753,8 +676,8 @@ job: **Additional details**: -Scripts you specify in `before_script` are concatenated with any scripts you specify -in the main [`script`](#script). The combined scripts execute together in a single shell. +- Scripts you specify in `before_script` are concatenated with any scripts you specify + in the main [`script`](#script). The combined scripts execute together in a single shell. **Related topics**: @@ -771,13 +694,13 @@ in the main [`script`](#script). The combined scripts execute together in a sing Use `after_script` to define an array of commands that run after each job, including failed jobs. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#custom-default-keyword-values). +[`default:` section](#default). **Possible inputs**: An array including: - Single line commands. - Long commands [split over multiple lines](script.md#split-long-commands). -- [YAML anchors](#yaml-anchors-for-scripts). +- [YAML anchors](yaml_optimization.md#yaml-anchors-for-scripts). **Example of `after_script`:** @@ -794,7 +717,7 @@ job: Scripts you specify in `after_script` execute in a new shell, separate from any `before_script` or `script` commands. As a result, they: -- Have the current working directory set back to the default (according to the [variables which define how the runner processes Git requests](#configure-runner-behavior-with-variables)). +- Have the current working directory set back to the default (according to the [variables which define how the runner processes Git requests](../runners/configure_runners.md#configure-runner-behavior-with-variables)). - Don't have access to changes done by commands defined in the `before_script` or `script`, including: - Command aliases and variables exported in `script` scripts. @@ -872,7 +795,7 @@ job4: Use the `.pre` stage to make a job run at the start of a pipeline. `.pre` is always the first stage in a pipeline. User-defined stages execute after `.pre`. -You do not need to define `.pre` in [`stages`](#stages). +You do not have to define `.pre` in [`stages`](#stages). You must have a job in at least one stage other than `.pre` or `.post`. @@ -907,7 +830,7 @@ job2: Use the `.post` stage to make a job run at the end of a pipeline. `.post` is always the last stage in a pipeline. User-defined stages execute before `.post`. -You do not need to define `.post` in [`stages`](#stages). +You do not have to define `.post` in [`stages`](#stages). You must have a job in at least one stage other than `.pre` or `.post`. @@ -938,18 +861,17 @@ job2: ### `extends` -> Introduced in GitLab 11.3. +Use `extends` to reuse configuration sections. It's an alternative to [YAML anchors](yaml_optimization.md#anchors) +and is a little more flexible and readable. -Use `extends` to reuse configuration sections. It's an alternative to [YAML anchors](#anchors) -and is a little more flexible and readable. You can use `extends` to reuse configuration -from [included configuration files](#use-extends-and-include-together). +**Keyword type**: Job keyword. You can use it only as part of a job. -In the following example, the `rspec` job uses the configuration from the `.tests` template job. -GitLab: +**Possible inputs:** -- Performs a reverse deep merge based on the keys. -- Merges the `.tests` content with the `rspec` job. -- Doesn't merge the values of the keys. +- The name of another job in the pipeline. +- A list (array) of names of other jobs in the pipeline. + +**Example of `extends`:** ```yaml .tests: @@ -967,6 +889,13 @@ rspec: - $RSPEC ``` +In this example, the `rspec` job uses the configuration from the `.tests` template job. +When creating the pipeline, GitLab: + +- Performs a reverse deep merge based on the keys. +- Merges the `.tests` content with the `rspec` job. +- Doesn't merge the values of the keys. + The result is this `rspec` job: ```yaml @@ -980,127 +909,18 @@ rspec: - $RSPEC ``` -`.tests` in this example is a [hidden job](#hide-jobs), but it's -possible to extend configuration from regular jobs as well. - -`extends` supports multi-level inheritance. You should avoid using more than three levels, -but you can use as many as eleven. The following example has two levels of inheritance: - -```yaml -.tests: - rules: - - if: $CI_PIPELINE_SOURCE == "push" - -.rspec: - extends: .tests - script: rake rspec - -rspec 1: - variables: - RSPEC_SUITE: '1' - extends: .rspec - -rspec 2: - variables: - RSPEC_SUITE: '2' - extends: .rspec - -spinach: - extends: .tests - script: rake spinach -``` - -In GitLab 12.0 and later, it's also possible to use multiple parents for -`extends`. - -#### Merge details - -You can use `extends` to merge hashes but not arrays. -The algorithm used for merge is "closest scope wins," so -keys from the last member always override anything defined on other -levels. For example: - -```yaml -.only-important: - variables: - URL: "http://my-url.internal" - IMPORTANT_VAR: "the details" - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - - if: $CI_COMMIT_BRANCH == "stable" - tags: - - production - script: - - echo "Hello world!" - -.in-docker: - variables: - URL: "http://docker-url.internal" - tags: - - docker - image: alpine - -rspec: - variables: - GITLAB: "is-awesome" - extends: - - .only-important - - .in-docker - script: - - rake rspec -``` - -The result is this `rspec` job: - -```yaml -rspec: - variables: - URL: "http://docker-url.internal" - IMPORTANT_VAR: "the details" - GITLAB: "is-awesome" - rules: - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH - - if: $CI_COMMIT_BRANCH == "stable" - tags: - - docker - image: alpine - script: - - rake rspec -``` - -In this example: - -- The `variables` sections merge, but `URL: "http://docker-url.internal"` overwrites `URL: "http://my-url.internal"`. -- `tags: ['docker']` overwrites `tags: ['production']`. -- `script` does not merge, but `script: ['rake rspec']` overwrites - `script: ['echo "Hello world!"']`. You can use [YAML anchors](#anchors) to merge arrays. - -#### Use `extends` and `include` together - -To reuse configuration from different configuration files, -combine `extends` and [`include`](#include). - -In the following example, a `script` is defined in the `included.yml` file. -Then, in the `.gitlab-ci.yml` file, `extends` refers -to the contents of the `script`: - -- `included.yml`: - - ```yaml - .template: - script: - - echo Hello! - ``` +**Additional details:** -- `.gitlab-ci.yml`: +- In GitLab 12.0 and later, you can use multiple parents for `extends`. +- The `extends` keyword supports up to eleven levels of inheritance, but you should + avoid using more than three levels. +- In the example above, `.tests` is a [hidden job](../jobs/index.md#hide-jobs), + but you can extend configuration from regular jobs as well. - ```yaml - include: included.yml +**Related topics:** - useTemplate: - image: alpine - extends: .template - ``` +- [Reuse configuration sections by using `extends`](yaml_optimization.md#use-extends-to-reuse-configuration-sections). +- Use `extends` to reuse configuration from [included configuration files](yaml_optimization.md#use-extends-and-include-together). ### `rules` @@ -1108,8 +928,11 @@ to the contents of the `script`: Use `rules` to include or exclude jobs in pipelines. -Rules are evaluated *in order* until the first match. When a match is found, the job -is either included or excluded from the pipeline, depending on the configuration. +Rules are evaluated when the pipeline is created, and evaluated *in order* +until the first match. When a match is found, the job is either included or excluded from the pipeline, +depending on the configuration. + +You cannot use dotenv variables created in job scripts in rules, because rules are evaluated before any jobs run. `rules` replaces [`only/except`](#only--except) and they can't be used together in the same job. If you configure one job to use both keywords, the GitLab returns @@ -1137,7 +960,7 @@ The job is not added to the pipeline: - If no rules match. - If a rule matches and has `when: never`. -You can use [`!reference` tags](#reference-tags) to [reuse `rules` configuration](../jobs/job_control.md#reuse-rules-in-different-jobs) +You can use [`!reference` tags](yaml_optimization.md#reference-tags) to [reuse `rules` configuration](../jobs/job_control.md#reuse-rules-in-different-jobs) in different jobs. #### `rules:if` @@ -1178,7 +1001,7 @@ job: all rules. You can't mix `when` at the job-level with `when` in rules. - Unlike variables in [`script`](../variables/index.md#use-cicd-variables-in-job-scripts) sections, variables in rules expressions are always formatted as `$VARIABLE`. - - You can use `rules:if` with `include` to [conditionally include other configuration files](#rules-with-include). + - You can use `rules:if` with `include` to [conditionally include other configuration files](includes.md#use-rules-with-include). **Related topics**: @@ -1225,6 +1048,7 @@ docker build: - `rules: changes` works the same way as [`only: changes` and `except: changes`](#onlychanges--exceptchanges). - You can use `when: never` to implement a rule similar to [`except:changes`](#onlychanges--exceptchanges). +- `changes` resolves to `true` if any of the matching files are changed (an `OR` operation). #### `rules:exists` @@ -1257,6 +1081,7 @@ job: file paths. After the 10,000th check, rules with patterned globs always match. In other words, the `exists` rule always assumes a match in projects with more than 10,000 files. +- `exists` resolves to `true` if any of the listed files are found (an `OR` operation). #### `rules:allow_failure` @@ -1369,7 +1194,7 @@ pipeline based on branch names or pipeline types. | `schedules` | For [scheduled pipelines](../pipelines/schedules.md). | | `tags` | When the Git reference for a pipeline is a tag. | | `triggers` | For pipelines created by using a [trigger token](../triggers/index.md#authentication-tokens). | - | `web` | For pipelines created by using **Run pipeline** button in the GitLab UI, from the project's **CI/CD > Pipelines** section. | + | `web` | For pipelines created by selecting **Run pipeline** in the GitLab UI, from the project's **CI/CD > Pipelines** section. | **Example of `only:refs` and `except:refs`**: @@ -1453,8 +1278,6 @@ deploy: #### `only:changes` / `except:changes` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/19232) in GitLab 11.4. - Use the `changes` keyword with `only` to run a job, or with `except` to skip a job, when a Git push event modifies a file. @@ -1489,10 +1312,12 @@ docker build: - docker/scripts/* - dockerfiles/**/* - more_scripts/*.{rb,py,sh} + - "**/*.json" ``` **Additional details**: +- `changes` resolves to `true` if any of the matching files are changed (an `OR` operation). - If you use refs other than `branches`, `external_pull_requests`, or `merge_requests`, `changes` can't determine if a given file is new or old and always returns `true`. - If you use `only: changes` with other refs, jobs ignore the changes and always run. @@ -1539,16 +1364,14 @@ that use `needs` can be visualized as a [directed acyclic graph](../directed_acy You can ignore stage ordering and run some jobs without waiting for others to complete. Jobs in multiple stages can run concurrently. -The following example creates four paths of execution: +**Keyword type**: Job keyword. You can use it only as part of a job. -- Linter: the `lint` job runs immediately without waiting for the `build` stage - to complete because it has no needs (`needs: []`). -- Linux path: the `linux:rspec` and `linux:rubocop` jobs runs as soon as the `linux:build` - job finishes without waiting for `mac:build` to finish. -- macOS path: the `mac:rspec` and `mac:rubocop` jobs runs as soon as the `mac:build` - job finishes, without waiting for `linux:build` to finish. -- The `production` job runs as soon as all previous jobs finish; in this case: - `linux:build`, `linux:rspec`, `linux:rubocop`, `mac:build`, `mac:rspec`, `mac:rubocop`. +**Possible inputs**: + +- An array of jobs. +- An empty array (`[]`), to set the job to start as soon as the pipeline is created. + +**Example of `needs`**: ```yaml linux:build: @@ -1569,32 +1392,33 @@ linux:rspec: needs: ["linux:build"] script: echo "Running rspec on linux..." -linux:rubocop: - stage: test - needs: ["linux:build"] - script: echo "Running rubocop on linux..." - mac:rspec: stage: test needs: ["mac:build"] script: echo "Running rspec on mac..." -mac:rubocop: - stage: test - needs: ["mac:build"] - script: echo "Running rubocop on mac..." - production: stage: deploy script: echo "Running production..." ``` -#### Requirements and limitations +This example creates four paths of execution: + +- Linter: The `lint` job runs immediately without waiting for the `build` stage + to complete because it has no needs (`needs: []`). +- Linux path: The `linux:rspec` job runs as soon as the `linux:build` + job finishes, without waiting for `mac:build` to finish. +- macOS path: The `mac:rspec` jobs runs as soon as the `mac:build` + job finishes, without waiting for `linux:build` to finish. +- The `production` job runs as soon as all previous jobs finish: + `linux:build`, `linux:rspec`, `mac:build`, `mac:rspec`. + +**Additional details**: -- The maximum number of jobs that a single job can need in the `needs:` array is limited: +- The maximum number of jobs that a single job can have in the `needs:` array is limited: - For GitLab.com, the limit is 50. For more information, see our [infrastructure issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/7541). - - For self-managed instances, the default limit is 50. This limit [can be changed](#changing-the-needs-job-limit). + - For self-managed instances, the default limit is 50. This limit [can be changed](../../administration/cicd.md#set-the-needs-job-limit). - If `needs:` refers to a job that uses the [`parallel`](#parallel) keyword, it depends on all jobs created in parallel, not just one job. It also downloads artifacts from all the parallel jobs by default. If the artifacts have the same @@ -1609,20 +1433,7 @@ production: - In GitLab 13.9 and older, if `needs:` refers to a job that might not be added to a pipeline because of `only`, `except`, or `rules`, the pipeline might fail to create. -##### Changing the `needs:` job limit **(FREE SELF)** - -The maximum number of jobs that can be defined in `needs:` defaults to 50. - -A GitLab administrator with [access to the GitLab Rails console](../../administration/feature_flags.md) -can choose a custom limit. For example, to set the limit to 100: - -```ruby -Plan.default.actual_limits.update!(ci_needs_size_limit: 100) -``` - -To disable directed acyclic graphs (DAG), set the limit to `0`. - -#### Artifact downloads with `needs` +#### `needs:artifacts` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.6. @@ -1633,84 +1444,72 @@ by default, because jobs with `needs` can start before earlier stages complete. Use `artifacts: true` (default) or `artifacts: false` to control when artifacts are downloaded in jobs that use `needs`. -In the following example, the `rspec` job downloads the `build_job` artifacts, but the -`rubocop` job does not: +**Keyword type**: Job keyword. You can use it only as part of a job. Must be used with `needs:job`. -```yaml -build_job: - stage: build - artifacts: - paths: - - binaries/ +**Possible inputs**: -rspec: +- `true` (default) or `false`. + +**Example of `needs:artifacts`**: + +```yaml +test-job1: stage: test needs: - - job: build_job + - job: build_job1 artifacts: true -rubocop: +test-job2: stage: test needs: - - job: build_job + - job: build_job2 artifacts: false -``` - -In the following example, the `rspec` job downloads the artifacts from all three `build_jobs`. -`artifacts` is: -- Set to true for `build_job_1`. -- Defaults to true for both `build_job_2` and `build_job_3`. - -```yaml -rspec: +test-job3: needs: - - job: build_job_1 + - job: build_job1 artifacts: true - - job: build_job_2 - - build_job_3 + - job: build_job2 + - build_job3 ``` -In GitLab 12.6 and later, you can't combine the [`dependencies`](#dependencies) keyword -with `needs`. +In this example: + +- The `test-job1` job downloads the `build_job1` artifacts +- The `test-job2` job does not download the `build_job2` artifacts. +- The `test-job3` job downloads the artifacts from all three `build_jobs`, because + `artifacts:` is `true`, or defaults to `true`, for all three needed jobs. -#### Cross project artifact downloads with `needs` **(PREMIUM)** +**Additional details**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.7. +- In GitLab 12.6 and later, you can't combine the [`dependencies`](#dependencies) keyword + with `needs`. -Use `needs` to download artifacts from up to five jobs in pipelines: +#### `needs:project` **(PREMIUM)** -- [On other refs in the same project](#artifact-downloads-between-pipelines-in-the-same-project). -- In different projects, groups and namespaces. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14311) in GitLab 12.7. -```yaml -build_job: - stage: build - script: - - ls -lhR - needs: - - project: namespace/group/project-name - job: build-1 - ref: main - artifacts: true -``` +Use `needs:project` to download artifacts from up to five jobs in other pipelines. +The artifacts are downloaded from the latest successful pipeline for the specified ref. -`build_job` downloads the artifacts from the latest successful `build-1` job -on the `main` branch in the `group/project-name` project. If the project is in the -same group or namespace, you can omit them from the `project:` keyword. For example, -`project: group/project-name` or `project: project-name`. +If there is a pipeline running for the specified ref, a job with `needs:project` +does not wait for the pipeline to complete. Instead, the job downloads the artifact +from the latest pipeline that completed successfully. -The user running the pipeline must have at least `reporter` access to the group or project, or the group/project must have public visibility. +`needs:project` must be used with `job:`, `ref:`, and `artifacts:`. -You cannot use cross project artifact downloads in the same job as [`trigger`](#trigger). +**Keyword type**: Job keyword. You can use it only as part of a job. -##### Artifact downloads between pipelines in the same project +**Possible inputs**: -Use `needs` to download artifacts from different pipelines in the current project. -Set the `project` keyword as the current project's name, and specify a ref. +- `needs:project`: A full project path, including namespace and group. If the + project is in the same group or namespace, you can omit them from the `project:` + keyword. For example: `project: group/project-name` or `project: project-name`. +- `job`: The job to download artifacts from. +- `ref`: The ref to download artifacts from. +- `artifacts`: Must be `true` to download artifacts. -In the following example, `build_job` downloads the artifacts for the latest successful -`build-1` job with the `other-ref` ref: +**Examples of `needs:project`**: ```yaml build_job: @@ -1718,16 +1517,17 @@ build_job: script: - ls -lhR needs: - - project: group/same-project-name + - project: namespace/group/project-name job: build-1 - ref: other-ref + ref: main artifacts: true ``` -CI/CD variable support for `project:`, `job:`, and `ref` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) -in GitLab 13.3. [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4. +In this example, `build_job` downloads the artifacts from the latest successful `build-1` job +on the `main` branch in the `group/project-name` project. -For example: +In GitLab 13.3 and later, you can use [CI/CD variables](../variables/index.md) in `needs:project`, +for example: ```yaml build_job: @@ -1741,57 +1541,83 @@ build_job: artifacts: true ``` -You can't download artifacts from jobs that run in [`parallel:`](#parallel). +**Additional details**: -To download artifacts between [parent-child pipelines](../pipelines/parent_child_pipelines.md), -use [`needs:pipeline`](#artifact-downloads-to-child-pipelines). +- To download artifacts from a different pipeline in the current project, set `project:` + to be the same as the current project, but use a different ref than the current pipeline. + Concurrent pipelines running on the same ref could override the artifacts. +- The user running the pipeline must have at least the Reporter role for the group or project, + or the group/project must have public visibility. +- You can't use `needs:project` in the same job as [`trigger`](#trigger). +- When using `needs:project` to download artifacts from another pipeline, the job does not wait for + the needed job to complete. [Directed acyclic graph](../directed_acyclic_graph/index.md) + behavior is limited to jobs in the same pipeline. Make sure that the needed job in the other + pipeline completes before the job that needs it tries to download the artifacts. +- You can't download artifacts from jobs that run in [`parallel:`](#parallel). +- Support for [CI/CD variables](../variables/index.md) in `project`, `job`, and `ref` was + [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/202093) in GitLab 13.3. + [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235761) in GitLab 13.4. -You should not download artifacts from the same ref as a running pipeline. Concurrent -pipelines running on the same ref could override the artifacts. +**Related topics**: -#### Artifact downloads to child pipelines +- To download artifacts between [parent-child pipelines](../pipelines/parent_child_pipelines.md), + use [`needs:pipeline:job`](#needspipelinejob). + +#### `needs:pipeline:job` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/255983) in GitLab 13.7. A [child pipeline](../pipelines/parent_child_pipelines.md) can download artifacts from a job in its parent pipeline or another child pipeline in the same parent-child pipeline hierarchy. -For example, with the following parent pipeline that has a job that creates some artifacts: +**Keyword type**: Job keyword. You can use it only as part of a job. -```yaml -create-artifact: - stage: build - script: echo 'sample artifact' > artifact.txt - artifacts: - paths: [artifact.txt] +**Possible inputs**: -child-pipeline: - stage: test - trigger: - include: child.yml - strategy: depend - variables: - PARENT_PIPELINE_ID: $CI_PIPELINE_ID -``` +- `needs:pipeline`: A pipeline ID. Must be a pipeline present in the same parent-child pipeline hierarchy. +- `job:`: The job to download artifacts from. -A job in the child pipeline can download artifacts from the `create-artifact` job in -the parent pipeline: +**Example of `needs:pipeline:job`**: -```yaml -use-artifact: - script: cat artifact.txt - needs: - - pipeline: $PARENT_PIPELINE_ID - job: create-artifact -``` +- Parent pipeline (`.gitlab-ci.yml`): -The `pipeline` attribute accepts a pipeline ID and it must be a pipeline present -in the same parent-child pipeline hierarchy of the given pipeline. + ```yaml + create-artifact: + stage: build + script: echo 'sample artifact' > artifact.txt + artifacts: + paths: [artifact.txt] -The `pipeline` attribute does not accept the current pipeline ID (`$CI_PIPELINE_ID`). -To download artifacts from a job in the current pipeline, use the basic form of [`needs`](#artifact-downloads-with-needs). + child-pipeline: + stage: test + trigger: + include: child.yml + strategy: depend + variables: + PARENT_PIPELINE_ID: $CI_PIPELINE_ID + ``` -#### Optional `needs` +- Child pipeline (`child.yml`): + + ```yaml + use-artifact: + script: cat artifact.txt + needs: + - pipeline: $PARENT_PIPELINE_ID + job: create-artifact + ``` + +In this example, the `create-artifact` job in the parent pipeline creates some artifacts. +The `child-pipeline` job triggers a child pipeline, and passes the `CI_PIPELINE_ID` +variable to the child pipeline as a new `PARENT_PIPELINE_ID` variable. The child pipeline +can use that variable in `needs:pipeline` to download artifacts from the parent pipeline. + +**Additional details**: + +- The `pipeline` attribute does not accept the current pipeline ID (`$CI_PIPELINE_ID`). + To download artifacts from a job in the current pipeline, use [`needs`](#needsartifacts). + +#### `needs:optional` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30680) in GitLab 13.10. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323891) in GitLab 14.0. @@ -1800,20 +1626,21 @@ To need a job that sometimes does not exist in the pipeline, add `optional: true to the `needs` configuration. If not defined, `optional: false` is the default. Jobs that use [`rules`](#rules), [`only`, or `except`](#only--except), might -not always exist in a pipeline. When the pipeline starts, it checks the `needs` -relationships before running. Without `optional: true`, needs relationships that +not always exist in a pipeline. When the pipeline is created, GitLab checks the `needs` +relationships before starting it. Without `optional: true`, needs relationships that point to a job that does not exist stops the pipeline from starting and causes a pipeline error similar to: - `'job1' job needs 'job2' job, but it was not added to the pipeline` -In this example: +**Keyword type**: Job keyword. You can use it only as part of a job. -- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec` - job waits for it to complete before starting. -- When the branch is not the default branch, the `build` job does not exist in the pipeline. - The `rspec` job runs immediately (similar to `needs: []`) because its `needs` - relationship to the `build` job is optional. +**Possible inputs**: + +- `job:`: The job to make optional. +- `true` or `false` (default). + +**Example of `needs:optional`**: ```yaml build: @@ -1828,6 +1655,42 @@ rspec: optional: true ``` +In this example: + +- When the branch is the default branch, the `build` job exists in the pipeline, and the `rspec` + job waits for it to complete before starting. +- When the branch is not the default branch, the `build` job does not exist in the pipeline. + The `rspec` job runs immediately (similar to `needs: []`) because its `needs` + relationship to the `build` job is optional. + +#### `needs:pipeline` + +You can mirror the pipeline status from an upstream pipeline to a bridge job by +using the `needs:pipeline` keyword. The latest pipeline status from the default branch is +replicated to the bridge job. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- A full project path, including namespace and group. If the + project is in the same group or namespace, you can omit them from the `project:` + keyword. For example: `project: group/project-name` or `project: project-name`. + +**Example of `needs:pipeline`**: + +```yaml +upstream_bridge: + stage: test + needs: + pipeline: other/project +``` + +**Additional details**: + +- If you add the `job` keyword to `needs:pipeline`, the job no longer mirrors the + pipeline status. The behavior changes to [`needs:pipeline:job`](#needspipelinejob). + ### `tags` > - A limit of 50 tags per job [enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/338929) in GitLab 14.3. @@ -1841,7 +1704,7 @@ example `ruby`, `postgres`, or `development`. To pick up and run a job, a runner be assigned every tag listed in the job. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#custom-default-keyword-values). +[`default:` section](#default). **Possible inputs**: @@ -2043,7 +1906,17 @@ In this example, the script: ### `environment` Use `environment` to define the [environment](../environments/index.md) that a job deploys to. -For example: + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: The name of the environment the job deploys to, in one of these +formats: + +- Plain text, including letters, digits, spaces, and these characters: `-`, `_`, `/`, `$`, `{`, `}`. +- CI/CD variables, including predefined, secure, or variables defined in the + `.gitlab-ci.yml` file. You can't use variables defined in a `script` section. + +**Example of `environment`**: ```yaml deploy to production: @@ -2052,20 +1925,27 @@ deploy to production: environment: production ``` -You can assign a value to the `environment` keyword by using: +**Additional details**: -- Plain text, like `production`. -- Variables, including CI/CD variables, predefined, secure, or variables - defined in the `.gitlab-ci.yml` file. +- If you specify an `environment` and no environment with that name exists, an environment is + created. -You can't use variables defined in a `script` section. +#### `environment:name` -If you specify an `environment` and no environment with that name exists, -an environment is created. +Set a name for an [environment](../environments/index.md). -#### `environment:name` +Common environment names are `qa`, `staging`, and `production`, but you can use any name. -Set a name for an [environment](../environments/index.md). For example: +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: The name of the environment the job deploys to, in one of these +formats: + +- Plain text, including letters, digits, spaces, and these characters: `-`, `_`, `/`, `$`, `{`, `}`. +- CI/CD variables, including predefined, secure, or variables defined in the + `.gitlab-ci.yml` file. You can't use variables defined in a `script` section. + +**Example of `environment:name`**: ```yaml deploy to production: @@ -2075,32 +1955,19 @@ deploy to production: name: production ``` -Common environment names are `qa`, `staging`, and `production`, but you can use any -name you want. - -You can assign a value to the `name` keyword by using: - -- Plain text, like `staging`. -- Variables, including CI/CD variables, predefined, secure, or variables - defined in the `.gitlab-ci.yml` file. +#### `environment:url` -You can't use variables defined in a `script` section. +Set a URL for an [environment](../environments/index.md). -The environment `name` can contain: +**Keyword type**: Job keyword. You can use it only as part of a job. -- Letters -- Digits -- Spaces -- `-` -- `_` -- `/` -- `$` -- `{` -- `}` +**Possible inputs**: A single URL, in one of these formats: -#### `environment:url` +- Plain text, like `https://prod.example.com`. +- CI/CD variables, including predefined, secure, or variables defined in the + `.gitlab-ci.yml` file. You can't use variables defined in a `script` section. -Set a URL for an [environment](../environments/index.md). For example: +**Example of `environment:url`**: ```yaml deploy to production: @@ -2111,16 +1978,10 @@ deploy to production: url: https://prod.example.com ``` -After the job completes, you can access the URL by using a button in the merge request, -environment, or deployment pages. - -You can assign a value to the `url` keyword by using: - -- Plain text, like `https://prod.example.com`. -- Variables, including CI/CD variables, predefined, secure, or variables - defined in the `.gitlab-ci.yml` file. +**Additional details**: -You can't use variables defined in a `script` section. +- After the job completes, you can access the URL by selecting a button in the merge request, + environment, or deployment pages. #### `environment:on_stop` @@ -2128,7 +1989,11 @@ Closing (stopping) environments can be achieved with the `on_stop` keyword defined under `environment`. It declares a different job that runs to close the environment. -Read the `environment:action` section for an example. +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Additional details**: + +- See [`environment:action`](#environmentaction) for more details and an example. #### `environment:action` @@ -2174,7 +2039,7 @@ Also in the example, `GIT_STRATEGY` is set to `none`. If the the runner won't try to check out the code after the branch is deleted. The example also overwrites global variables. If your `stop` `environment` job depends -on global variables, use [anchor variables](#yaml-anchors-for-variables) when you set the `GIT_STRATEGY` +on global variables, use [anchor variables](yaml_optimization.md#yaml-anchors-for-variables) when you set the `GIT_STRATEGY` to change the job without overriding the global variables. The `stop_review_app` job is **required** to have the following keywords defined: @@ -2199,10 +2064,19 @@ In the examples above, if the configuration is not identical: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20956) in GitLab 12.8. -The `auto_stop_in` keyword is for specifying the lifetime of the environment, -that when expired, GitLab automatically stops them. +The `auto_stop_in` keyword specifies the lifetime of the environment. When an environment expires, GitLab +automatically stops it. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: A period of time written in natural language. For example, +these are all equivalent: + +- `168 hours` +- `7 days` +- `one week` -For example, +**Example of `environment:auto_stop_in`**: ```yaml review_app: @@ -2215,8 +2089,9 @@ review_app: When the environment for `review_app` is created, the environment's lifetime is set to `1 day`. Every time the review app is deployed, that lifetime is also reset to `1 day`. -For more information, see -[the environments auto-stop documentation](../environments/index.md#stop-an-environment-after-a-certain-time-period) +**Related topics**: + +- [Environments auto-stop documentation](../environments/index.md#stop-an-environment-after-a-certain-time-period). #### `environment:kubernetes` @@ -2225,7 +2100,9 @@ For more information, see Use the `kubernetes` keyword to configure deployments to a [Kubernetes cluster](../../user/infrastructure/clusters/index.md) that is associated with your project. -For example: +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Example of `environment:kubernetes`**: ```yaml deploy: @@ -2241,20 +2118,34 @@ This configuration sets up the `deploy` job to deploy to the `production` environment, using the `production` [Kubernetes namespace](https://kubernetes.io/docs/concepts/overview/working-with-objects/namespaces/). -For more information, see -[Available settings for `kubernetes`](../environments/index.md#configure-kubernetes-deployments). +**Additional details**: -NOTE: -Kubernetes configuration is not supported for Kubernetes clusters -that are [managed by GitLab](../../user/project/clusters/gitlab_managed_clusters.md). -To follow progress on support for GitLab-managed clusters, see the -[relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). +- Kubernetes configuration is not supported for Kubernetes clusters + that are [managed by GitLab](../../user/project/clusters/gitlab_managed_clusters.md). + To follow progress on support for GitLab-managed clusters, see the + [relevant issue](https://gitlab.com/gitlab-org/gitlab/-/issues/38054). + +**Related topics**: + +- [Available settings for `kubernetes`](../environments/index.md#configure-kubernetes-deployments-deprecated). #### `environment:deployment_tier` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300741) in GitLab 13.10. -Use the `deployment_tier` keyword to specify the tier of the deployment environment: +Use the `deployment_tier` keyword to specify the tier of the deployment environment. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: One of the following: + +- `production` +- `staging` +- `testing` +- `development` +- `other` + +**Example of `environment:deployment_tier`**: ```yaml deploy: @@ -2264,8 +2155,9 @@ deploy: deployment_tier: production ``` -For more information, -see [Deployment tier of environments](../environments/index.md#deployment-tier-of-environments). +**Related topics**: + +- [Deployment tier of environments](../environments/index.md#deployment-tier-of-environments). #### Dynamic environments @@ -2306,7 +2198,8 @@ Learn more about caches in [Caching in GitLab CI/CD](../caching/index.md). Use the `cache:paths` keyword to choose which files or directories to cache. -**Keyword type**: Job-specific. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). **Possible inputs**: An array of paths relative to the project directory (`$CI_PROJECT_DIR`). You can use wildcards that use [glob](https://en.wikipedia.org/wiki/Glob_(programming)) @@ -2345,7 +2238,8 @@ that use the same cache key use the same cache, including in different pipelines If not set, the default key is `default`. All jobs with the `cache:` keyword but no `cache:key` share the `default` cache. -**Keyword type**: Job-specific. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). **Possible inputs**: @@ -2367,7 +2261,7 @@ cache-job: **Additional details**: -- If you use **Windows Batch** to run your shell scripts you need to replace +- If you use **Windows Batch** to run your shell scripts you must replace `$` with `%`. For example: `key: %CI_COMMIT_REF_SLUG%` - The `cache:key` value can't contain: @@ -2394,7 +2288,8 @@ Use the `cache:key:files` keyword to generate a new key when one or two specific change. `cache:key:files` lets you reuse some caches, and rebuild them less often, which speeds up subsequent pipeline runs. -**Keyword type**: Job-specific. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). **Possible inputs**: An array of one or two file paths. @@ -2420,9 +2315,11 @@ these files changes, a new cache key is computed and a new cache is created. Any job runs that use the same `Gemfile.lock` and `package.json` with `cache:key:files` use the new cache, instead of rebuilding the dependencies. -**Additional details**: The cache `key` is a SHA computed from the most recent commits -that changed each listed file. If neither file is changed in any commits, the -fallback key is `default`. +**Additional details**: + +- The cache `key` is a SHA computed from the most recent commits +that changed each listed file. + If neither file is changed in any commits, the fallback key is `default`. ##### `cache:key:prefix` @@ -2430,7 +2327,8 @@ fallback key is `default`. Use `cache:key:prefix` to combine a prefix with the SHA computed for [`cache:key:files`](#cachekeyfiles). -**Keyword type**: Job-specific. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). **Possible inputs**: @@ -2458,14 +2356,16 @@ If a branch changes `Gemfile.lock`, that branch has a new SHA checksum for `cach A new cache key is generated, and a new cache is created for that key. If `Gemfile.lock` is not found, the prefix is added to `default`, so the key in the example would be `rspec-default`. -**Additional details**: If no file in `cache:key:files` is changed in any commits, -the prefix is added to the `default` key. +**Additional details**: + +- If no file in `cache:key:files` is changed in any commits, the prefix is added to the `default` key. #### `cache:untracked` Use `untracked: true` to cache all files that are untracked in your Git repository: -**Keyword type**: Job-specific. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). **Possible inputs**: `true` or `false` (default). @@ -2498,7 +2398,8 @@ rspec: Use `cache:when` to define when to save the cache, based on the status of the job. -**Keyword type**: Job-specific. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). **Possible inputs**: @@ -2523,7 +2424,7 @@ This example stores the cache whether or not the job fails or succeeds. To change the upload and download behavior of a cache, use the `cache:policy` keyword. By default, the job downloads the cache when the job starts, and uploads changes -to the cache when the job ends. This is the `pull-push` policy (default). +to the cache when the job ends. This caching style is the `pull-push` policy (default). To set a job to only download the cache when the job starts, but never upload changes when the job finishes, use `cache:policy:pull`. @@ -2535,7 +2436,8 @@ Use the `pull` policy when you have many jobs executing in parallel that use the This policy speeds up job execution and reduces load on the cache server. You can use a job with the `push` policy to build the cache. -**Keyword type**: Job-specific. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). **Possible inputs**: @@ -2569,74 +2471,44 @@ faster-test-job: - echo "Running tests..." ``` -### `artifacts` - -Use `artifacts` to specify a list of files and directories that are -attached to the job when it [succeeds, fails, or always](#artifactswhen). - -The artifacts are sent to GitLab after the job finishes. They are -available for download in the GitLab UI if the size is not -larger than the [maximum artifact size](../../user/gitlab_com/index.md#gitlab-cicd). - -By default, jobs in later stages automatically download all the artifacts created -by jobs in earlier stages. You can control artifact download behavior in jobs with -[`dependencies`](#dependencies). - -When using the [`needs`](#artifact-downloads-with-needs) keyword, jobs can only download -artifacts from the jobs defined in the `needs` configuration. - -Job artifacts are only collected for successful jobs by default, and -artifacts are restored after [caches](#cache). - -[Read more about artifacts](../pipelines/job_artifacts.md). - -#### `dependencies` - -By default, all `artifacts` from previous stages -are passed to each job. However, you can use the `dependencies` keyword to -define a limited list of jobs to fetch artifacts from. You can also set a job to download no artifacts at all. +### `dependencies` -To use this feature, define `dependencies` in context of the job and pass -a list of all previous jobs the artifacts should be downloaded from. +Use the `dependencies` keyword to define a list of jobs to fetch [artifacts](#artifacts) from. +You can also set a job to download no artifacts at all. -You can define jobs from stages that were executed before the current one. -An error occurs if you define jobs from the current or an upcoming stage. +If you do not use `dependencies`, all artifacts from previous stages are passed to each job. -To prevent a job from downloading artifacts, define an empty array. +**Keyword type**: Job keyword. You can use it only as part of a job. -When you use `dependencies`, the status of the previous job is not considered. -If a job fails or it's a manual job that isn't triggered, no error occurs. +**Possible inputs**: -The following example defines two jobs with artifacts: `build:osx` and -`build:linux`. When the `test:osx` is executed, the artifacts from `build:osx` -are downloaded and extracted in the context of the build. The same happens -for `test:linux` and artifacts from `build:linux`. +- The names of jobs to fetch artifacts from. +- An empty array (`[]`), to configure the job to not download any artifacts. -The job `deploy` downloads artifacts from all previous jobs because of -the [stage](#stages) precedence: +**Example of `dependencies`**: ```yaml -build:osx: +build osx: stage: build script: make build:osx artifacts: paths: - binaries/ -build:linux: +build linux: stage: build script: make build:linux artifacts: paths: - binaries/ -test:osx: +test osx: stage: test script: make test:osx dependencies: - build:osx -test:linux: +test linux: stage: test script: make test:linux dependencies: @@ -2647,14 +2519,39 @@ deploy: script: make deploy ``` -##### When a dependent job fails +In this example, two jobs have artifacts: `build osx` and `build linux`. When `test osx` is executed, +the artifacts from `build osx` are downloaded and extracted in the context of the build. +The same thing happens for `test linux` and artifacts from `build linux`. -> Introduced in GitLab 10.3. +The `deploy` job downloads artifacts from all previous jobs because of +the [stage](#stages) precedence. -If the artifacts of the job that is set as a dependency are -[expired](#artifactsexpire_in) or -[deleted](../pipelines/job_artifacts.md#delete-job-artifacts), then -the dependent job fails. +**Additional details**: + +- The job status does not matter. If a job fails or it's a manual job that isn't triggered, no error occurs. +- If the artifacts of a dependent job are [expired](#artifactsexpire_in) or + [deleted](../pipelines/job_artifacts.md#delete-job-artifacts), then the job fails. + +### `artifacts` + +Use `artifacts` to specify a list of files and directories that are +attached to the job when it [succeeds, fails, or always](#artifactswhen). + +The artifacts are sent to GitLab after the job finishes. They are +available for download in the GitLab UI if the size is not +larger than the [maximum artifact size](../../user/gitlab_com/index.md#gitlab-cicd). + +By default, jobs in later stages automatically download all the artifacts created +by jobs in earlier stages. You can control artifact download behavior in jobs with +[`dependencies`](#dependencies). + +When using the [`needs`](#needs) keyword, jobs can only download +artifacts from the jobs defined in the `needs` configuration. + +Job artifacts are only collected for successful jobs by default, and +artifacts are restored after [caches](#cache). + +[Read more about artifacts](../pipelines/job_artifacts.md). #### `artifacts:exclude` @@ -2706,13 +2603,12 @@ Files matched by [`artifacts:untracked`](#artifactsuntracked) can be excluded us Use `expire_in` to specify how long [job artifacts](../pipelines/job_artifacts.md) are stored before they expire and are deleted. The `expire_in` setting does not affect: -- Artifacts from the latest job, unless this keeping the latest job artifacts is: +- Artifacts from the latest job, unless keeping the latest job artifacts is: - [Disabled at the project level](../pipelines/job_artifacts.md#keep-artifacts-from-most-recent-successful-jobs). - [Disabled instance-wide](../../user/admin_area/settings/continuous_integration.md#keep-the-latest-artifacts-for-all-jobs-in-the-latest-successful-pipelines). -- [Pipeline artifacts](../pipelines/pipeline_artifacts.md). It's not possible to specify an - expiration date for these: - - Pipeline artifacts from the latest pipeline are kept forever. - - Other pipeline artifacts are erased after one week. +- [Pipeline artifacts](../pipelines/pipeline_artifacts.md). You can't specify an expiration date for + pipeline artifacts. See [When pipeline artifacts are deleted](../pipelines/pipeline_artifacts.md#when-pipeline-artifacts-are-deleted) + for more information. The value of `expire_in` is an elapsed time in seconds, unless a unit is provided. Valid values include: @@ -2742,7 +2638,7 @@ time is not defined, it defaults to the To override the expiration date and protect artifacts from being automatically deleted: -- Use the **Keep** button on the job page. +- Select **Keep** on the job page. - [In GitLab 13.3 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/22761), set the value of `expire_in` to `never`. @@ -2847,7 +2743,7 @@ job: --- -If you use **Windows Batch** to run your shell scripts you need to replace +If you use **Windows Batch** to run your shell scripts you must replace `$` with `%`: ```yaml @@ -2858,7 +2754,7 @@ job: - binaries/ ``` -If you use **Windows PowerShell** to run your shell scripts you need to replace +If you use **Windows PowerShell** to run your shell scripts you must replace `$` with `$env:`: ```yaml @@ -2876,9 +2772,8 @@ link outside it. You can use Wildcards that use [glob](https://en.wikipedia.org/ patterns and: - In [GitLab Runner 13.0 and later](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/2620), -[`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). -- In GitLab Runner 12.10 and earlier, -[`filepath.Match`](https://pkg.go.dev/path/filepath#Match). + [`doublestar.Glob`](https://pkg.go.dev/github.com/bmatcuk/doublestar@v1.2.2?tab=doc#Match). +- In GitLab Runner 12.10 and earlier, [`filepath.Match`](https://pkg.go.dev/path/filepath#Match). To restrict which jobs a specific job fetches artifacts from, see [dependencies](#dependencies). @@ -2959,19 +2854,39 @@ artifacts: #### `artifacts:reports` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. -> - Requires GitLab Runner 11.2 and above. +Use [`artifacts:reports`](#artifactsreports) to: -Use [`artifacts:reports`](#artifactsreports) -to collect test reports, code quality reports, and security reports from jobs. -It also exposes these reports in the GitLab UI (merge requests, pipeline views, and security dashboards). +- Collect test reports, code quality reports, security reports, and other artifacts generated by included templates in + jobs. +- Some of these reports are used to display information in: + - Merge requests. + - Pipeline views. + - [Security dashboards](../../user/application_security/security_dashboard/index.md). The test reports are collected regardless of the job results (success or failure). You can use [`artifacts:expire_in`](#artifactsexpire_in) to set up an expiration date for their artifacts. -If you also want the ability to browse the report output files, include the -[`artifacts:paths`](#artifactspaths) keyword. +Some `artifacts:reports` types can be generated by multiple jobs in the same pipeline, and used by merge request or +pipeline features from each job. + +To be able to browse the report output files, include the [`artifacts:paths`](#artifactspaths) keyword. + +NOTE: +Combined reports in parent pipelines using [artifacts from child pipelines](#needspipelinejob) is +not supported. Track progress on adding support in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/215725). + +##### `artifacts:reports:accessibility` **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39425) in GitLab 12.8. + +The `accessibility` report uses [pa11y](https://pa11y.org/) to report on the accessibility impact +of changes introduced in merge requests. + +GitLab can display the results of one or more reports in the merge request +[accessibility widget](../../user/project/merge_requests/accessibility_testing.md#accessibility-merge-request-widget). + +For more information, see [Accessibility testing](../../user/project/merge_requests/accessibility_testing.md). ##### `artifacts:reports:api_fuzzing` **(ULTIMATE)** @@ -2981,100 +2896,128 @@ If you also want the ability to browse the report output files, include the The `api_fuzzing` report collects [API Fuzzing bugs](../../user/application_security/api_fuzzing/index.md) as artifacts. -The collected API Fuzzing report uploads to GitLab as an artifact and is summarized in merge -requests and the pipeline view. It's also used to provide data for security dashboards. +GitLab can display the results of one or more reports in: + +- The merge request [security widget](../../user/application_security/api_fuzzing/index.md#view-details-of-an-api-fuzzing-vulnerability). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [security dashboard](../../user/application_security/api_fuzzing/index.md#security-dashboard). + +##### `artifacts:reports:browser_performance` **(PREMIUM)** + +> [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. + +The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) +as artifacts. + +GitLab can display the results of one report in the merge request +[browser performance testing widget](../../user/project/merge_requests/browser_performance_testing.md#how-browser-performance-testing-works). + +GitLab cannot display the combined results of multiple `browser_performance` reports. + +##### `artifacts:reports:cluster_image_scanning` **(ULTIMATE)** + +> - Introduced in GitLab 14.1. +> - Requires GitLab Runner 14.1 and above. + +The `cluster_image_scanning` report collects `CLUSTER_IMAGE_SCANNING` vulnerabilities. The collected +`CLUSTER_IMAGE_SCANNING` report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: + +- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). ##### `artifacts:reports:cobertura` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. -> - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3708) in GitLab 12.9. The `cobertura` report collects [Cobertura coverage XML files](../../user/project/merge_requests/test_coverage_visualization.md). -The collected Cobertura coverage reports upload to GitLab as an artifact -and display in merge requests. +The collected Cobertura coverage reports upload to GitLab as an artifact. -Cobertura was originally developed for Java, but there are many -third party ports for other languages like JavaScript, Python, Ruby, and so on. +GitLab can display the results of one or more reports in the merge request +[diff annotations](../../user/project/merge_requests/test_coverage_visualization.md). + +Cobertura was originally developed for Java, but there are many third-party ports for other languages such as +JavaScript, Python, and Ruby. ##### `artifacts:reports:codequality` -> - Introduced in GitLab 11.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. -> - Requires GitLab Runner 11.5 and above. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212499) to GitLab Free in 13.2. -The `codequality` report collects [Code Quality issues](../../user/project/merge_requests/code_quality.md) -as artifacts. +The `codequality` report collects [code quality issues](../../user/project/merge_requests/code_quality.md). The +collected code quality report uploads to GitLab as an artifact. + +GitLab can display the results of: -The collected Code Quality report uploads to GitLab as an artifact and is summarized in merge requests. +- One or more reports in the merge request [code quality widget](../../user/project/merge_requests/code_quality.md#code-quality-widget). +- Only one report in: + - The merge request [diff annotations](../../user/project/merge_requests/code_quality.md#code-quality-in-diff-view). + Track progress on adding support for multiple reports in [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/328257). + - The [full report](../metrics_reports.md). Track progress on adding support for multiple reports in + [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/9014). ##### `artifacts:reports:container_scanning` **(ULTIMATE)** -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. +The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md). +The collected Container Scanning report uploads to GitLab as an artifact. -The `container_scanning` report collects [Container Scanning vulnerabilities](../../user/application_security/container_scanning/index.md) -as artifacts. +GitLab can display the results of one or more reports in: -The collected Container Scanning report uploads to GitLab as an artifact and -is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. +- The merge request [container scanning widget](../../user/application_security/container_scanning/index.md). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). ##### `artifacts:reports:coverage_fuzzing` **(ULTIMATE)** > - Introduced in GitLab 13.4. > - Requires GitLab Runner 13.4 or later. -The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md) -as artifacts. - -The collected coverage fuzzing report uploads to GitLab as an artifact and is summarized in merge -requests and the pipeline view. It's also used to provide data for security dashboards. +The `coverage_fuzzing` report collects [coverage fuzzing bugs](../../user/application_security/coverage_fuzzing/index.md). +The collected coverage fuzzing report uploads to GitLab as an artifact. +GitLab can display the results of one or more reports in: -##### `artifacts:reports:cluster_image_scanning` **(ULTIMATE)** - -> - Introduced in GitLab 14.1. -> - Requires GitLab Runner 14.1 and above. - -The `cluster_image_scanning` report collects `CLUSTER_IMAGE_SCANNING` vulnerabilities -as artifacts. - -The collected `CLUSTER_IMAGE_SCANNING` report uploads to GitLab as an artifact and -is summarized in the pipeline view. It's also used to provide data for security -dashboards. +- The merge request [coverage fuzzing widget](../../user/application_security/coverage_fuzzing/index.md#interacting-with-the-vulnerabilities). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). ##### `artifacts:reports:dast` **(ULTIMATE)** -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. +The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md). The collected DAST +report uploads to GitLab as an artifact. -The `dast` report collects [DAST vulnerabilities](../../user/application_security/dast/index.md) -as artifacts. +GitLab can display the results of one or more reports in: -The collected DAST report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. +- The merge request [security widget](../../user/application_security/dast/index.md#view-details-of-a-vulnerability-detected-by-dast). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). ##### `artifacts:reports:dependency_scanning` **(ULTIMATE)** -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. +The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md). +The collected Dependency Scanning report uploads to GitLab as an artifact. -The `dependency_scanning` report collects [Dependency Scanning vulnerabilities](../../user/application_security/dependency_scanning/index.md) -as artifacts. +GitLab can display the results of one or more reports in: -The collected Dependency Scanning report uploads to GitLab as an artifact and is summarized in merge requests and the pipeline view. It's also used to provide data for security -dashboards. +- The merge request [dependency scanning widget](../../user/application_security/dependency_scanning/index.md#overview). +- The pipeline [**Security** tab](../../user/application_security/security_dashboard/index.md#pipeline-security). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). +- The [Project Vulnerability report](../../user/application_security/vulnerability_report/index.md). +- The [dependency list](../../user/application_security/dependency_list/). ##### `artifacts:reports:dotenv` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. -> - Requires GitLab Runner 11.5 and later. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17066) in GitLab 12.9. The `dotenv` report collects a set of environment variables as artifacts. The collected variables are registered as runtime-created variables of the job, which is useful to [set dynamic environment URLs after a job finishes](../environments/index.md#set-dynamic-environment-urls-after-a-job-finishes). -There are a couple of exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv#rules): +The exceptions to the [original dotenv rules](https://github.com/motdotla/dotenv#rules) are: - The variable key can contain only letters, digits, and underscores (`_`). - The maximum size of the `.env` file is 5 KB. @@ -3088,13 +3031,9 @@ There are a couple of exceptions to the [original dotenv rules](https://github.c ##### `artifacts:reports:junit` -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/20390) in GitLab 11.2. -> - Requires GitLab Runner 11.2 and above. - -The `junit` report collects [JUnit report format XML files](https://www.ibm.com/docs/en/adfz/developer-for-zos/14.1.0?topic=formats-junit-xml-format) -as artifacts. Although JUnit was originally developed in Java, there are many -third party ports for other -languages like JavaScript, Python, Ruby, and so on. +The `junit` report collects [JUnit report format XML files](https://www.ibm.com/docs/en/adfz/developer-for-zos/14.1.0?topic=formats-junit-xml-format). +The collected Unit test reports upload to GitLab as an artifact. Although JUnit was originally developed in Java, there +are many third-party ports for other languages such as JavaScript, Python, and Ruby. See [Unit test reports](../unit_test_reports.md) for more details and examples. Below is an example of collecting a JUnit report format XML file from Ruby's RSpec test tool: @@ -3110,79 +3049,72 @@ rspec: junit: rspec.xml ``` -The collected Unit test reports upload to GitLab as an artifact and display in merge requests. +GitLab can display the results of one or more reports in: -If the JUnit tool you use exports to multiple XML files, specify -multiple test report paths within a single job to -concatenate them into a single file. Use a filename pattern (`junit: rspec-*.xml`), -an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`), or a -combination thereof (`junit: [rspec.xml, test-results/TEST-*.xml]`). +- The merge request [code quality widget](../../ci/unit_test_reports.md#how-it-works). +- The [full report](../../ci/unit_test_reports.md#viewing-unit-test-reports-on-gitlab). + +Some JUnit tools export to multiple XML files. You can specify multiple test report paths in a single job to +concatenate them into a single file. Use either: + +- A filename pattern (`junit: rspec-*.xml`). +- an array of filenames (`junit: [rspec-1.xml, rspec-2.xml, rspec-3.xml]`). +- A Combination of both (`junit: [rspec.xml, test-results/TEST-*.xml]`). ##### `artifacts:reports:license_scanning` **(ULTIMATE)** -> - Introduced in GitLab 12.8. -> - Requires GitLab Runner 11.5 and above. +> Introduced in GitLab 12.8. -The `license_scanning` report collects [Licenses](../../user/compliance/license_compliance/index.md) -as artifacts. +The License Compliance report collects [Licenses](../../user/compliance/license_compliance/index.md). The License +Compliance report uploads to GitLab as an artifact. -The License Compliance report uploads to GitLab as an artifact and displays automatically in merge requests and the pipeline view, and provide data for security -dashboards. +GitLab can display the results of one or more reports in: + +- The merge request [license compliance widget](../../user/compliance/license_compliance/index.md). +- The [license list](../../user/compliance/license_compliance/index.md#license-list). ##### `artifacts:reports:load_performance` **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35260) in GitLab 13.2. > - Requires GitLab Runner 11.5 and above. -The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md) -as artifacts. - -The report is uploaded to GitLab as an artifact and is -shown in merge requests automatically. +The `load_performance` report collects [Load Performance Testing metrics](../../user/project/merge_requests/load_performance_testing.md). +The report is uploaded to GitLab as an artifact. -##### `artifacts:reports:metrics` **(PREMIUM)** +GitLab can display the results of only one report in the merge request +[load testing widget](../../user/project/merge_requests/load_performance_testing.md#how-load-performance-testing-works). -> Introduced in GitLab 11.10. +GitLab cannot display the combined results of multiple `load_performance` reports. -The `metrics` report collects [Metrics](../metrics_reports.md) -as artifacts. - -The collected Metrics report uploads to GitLab as an artifact and displays in merge requests. - -##### `artifacts:reports:browser_performance` **(PREMIUM)** - -> - Introduced in GitLab 11.5. -> - Requires GitLab Runner 11.5 and above. -> - [Name changed](https://gitlab.com/gitlab-org/gitlab/-/issues/225914) from `artifacts:reports:performance` in GitLab 14.0. +##### `artifacts:reports:metrics` **(PREMIUM)** -The `browser_performance` report collects [Browser Performance Testing metrics](../../user/project/merge_requests/browser_performance_testing.md) -as artifacts. +The `metrics` report collects [Metrics](../metrics_reports.md). The collected Metrics report uploads to GitLab as an +artifact. -The collected Browser Performance report uploads to GitLab as an artifact and displays in merge requests. +GitLab can display the results of one or more reports in the merge request +[metrics reports widget](../../ci/metrics_reports.md#metrics-reports). ##### `artifacts:reports:requirements` **(ULTIMATE)** > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/2859) in GitLab 13.1. -> - Requires GitLab Runner 11.5 and above. -The `requirements` report collects `requirements.json` files as artifacts. +The `requirements` report collects `requirements.json` files. The collected Requirements report uploads to GitLab as an +artifact and existing [requirements](../../user/project/requirements/index.md) are marked as Satisfied. -The collected Requirements report uploads to GitLab as an artifact and -existing [requirements](../../user/project/requirements/index.md) are -marked as Satisfied. +GitLab can display the results of one or more reports in the +[project requirements](../../user/project/requirements/index.md#view-a-requirement). ##### `artifacts:reports:sast` -> - Introduced in GitLab 11.5. > - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) from GitLab Ultimate to GitLab Free in 13.3. -> - Requires GitLab Runner 11.5 and above. -The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md) -as artifacts. +The `sast` report collects [SAST vulnerabilities](../../user/application_security/sast/index.md). The collected SAST +report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in: -The collected SAST report uploads to GitLab as an artifact and is summarized -in merge requests and the pipeline view. It's also used to provide data for security -dashboards. +- The merge request [SAST widget](../../user/application_security/sast/index.md#static-application-security-testing-sast). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). ##### `artifacts:reports:secret_detection` @@ -3190,22 +3122,27 @@ dashboards. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) to GitLab Free in 13.3. > - Requires GitLab Runner 11.5 and above. -The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md) -as artifacts. +The `secret-detection` report collects [detected secrets](../../user/application_security/secret_detection/index.md). +The collected Secret Detection report is uploaded to GitLab. + +GitLab can display the results of one or more reports in: -The collected Secret Detection report is uploaded to GitLab as an artifact and summarized -in the merge requests and pipeline view. It's also used to provide data for security -dashboards. +- The merge request [secret scanning widget](../../user/application_security/secret_detection/index.md). +- The [pipeline **Security** tab](../../user/application_security/index.md#view-security-scan-information-in-the-pipeline-security-tab). +- The [security dashboard](../../user/application_security/security_dashboard/index.md). ##### `artifacts:reports:terraform` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207528) in GitLab 13.0. > - Requires [GitLab Runner](https://docs.gitlab.com/runner/) 11.5 and above. -The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/iac/mr_integration.md#configure-terraform-report-artifacts). The collected Terraform -plan report uploads to GitLab as an artifact and displays -in merge requests. For more information, see -[Output `terraform plan` information into a merge request](../../user/infrastructure/iac/mr_integration.md). +The `terraform` report obtains a Terraform `tfplan.json` file. [JQ processing required to remove credentials](../../user/infrastructure/iac/mr_integration.md#configure-terraform-report-artifacts). +The collected Terraform plan report uploads to GitLab as an artifact. + +GitLab can display the results of one or more reports in the merge request +[terraform widget](../../user/infrastructure/iac/mr_integration.md#output-terraform-plan-information-into-a-merge-request). + +For more information, see [Output `terraform plan` information into a merge request](../../user/infrastructure/iac/mr_integration.md). #### `artifacts:untracked` @@ -3247,7 +3184,7 @@ failure. 1. `on_success` (default): Upload artifacts only when the job succeeds. 1. `on_failure`: Upload artifacts only when the job fails. -1. `always`: Always upload artifacts. Useful, for example, when +1. `always`: Always upload artifacts. For example, when [uploading artifacts](../unit_test_reports.md#viewing-junit-screenshots-on-gitlab) required to troubleshoot failing tests. @@ -3339,8 +3276,6 @@ to select a specific site profile and scanner profile. ### `retry` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/3515) in GitLab 11.5, you can control which failures to retry on. - Use `retry` to configure how many times a job is retried if it fails. If not defined, defaults to `0` and jobs do not retry. @@ -3351,7 +3286,7 @@ By default, all failure types cause the job to be retried. Use [`retry:when`](#r to select which failures to retry on. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#custom-default-keyword-values). +[`default:` section](#default). **Possible inputs**: `0` (default), `1`, or `2`. @@ -3370,7 +3305,7 @@ Use `retry:when` with `retry:max` to retry jobs for only specific failure cases. `0`, `1`, or `2`. **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#custom-default-keyword-values). +[`default:` section](#default). **Possible inputs**: A single failure type, or an array of one or more failure types: @@ -3436,7 +3371,7 @@ The job-level timeout can be longer than the [project-level timeout](../pipeline but can't be longer than the [runner's timeout](../runners/configure_runners.md#set-maximum-job-timeout-for-a-runner). **Keyword type**: Job keyword. You can use it only as part of a job or in the -[`default:` section](#custom-default-keyword-values). +[`default:` section](#default). **Possible inputs**: A period of time written in natural language. For example, these are all equivalent: @@ -3458,13 +3393,17 @@ test: ### `parallel` -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/21480) in GitLab 11.5. +Use `parallel` to run a job multiple times in parallel in a single pipeline. + +Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently. + +Parallel jobs are named sequentially from `job_name 1/N` to `job_name N/N`. + +**Keyword type**: Job keyword. You can use it only as part of a job. -Use `parallel` to configure how many instances of a job to run in parallel. -The value can be from 2 to 50. +**Possible inputs**: A numeric value from `2` to `50`. -The `parallel` keyword creates N instances of the same job that run in parallel. -They are named sequentially from `job_name 1/N` to `job_name N/N`: +**Example of `parallel`**: ```yaml test: @@ -3472,47 +3411,32 @@ test: parallel: 5 ``` -Every parallel job has a `CI_NODE_INDEX` and `CI_NODE_TOTAL` -[predefined CI/CD variable](../variables/index.md#predefined-cicd-variables) set. +This example creates 5 jobs that run in parallel, named `test 1/5` to `test 5/5`. -Different languages and test suites have different methods to enable parallelization. -For example, use [Semaphore Test Boosters](https://github.com/renderedtext/test-boosters) -and RSpec to run Ruby tests in parallel: - -```ruby -# Gemfile -source 'https://rubygems.org' +**Additional details**: -gem 'rspec' -gem 'semaphore_test_boosters' -``` +- Every parallel job has a `CI_NODE_INDEX` and `CI_NODE_TOTAL` + [predefined CI/CD variable](../variables/index.md#predefined-cicd-variables) set. -```yaml -test: - parallel: 3 - script: - - bundle - - bundle exec rspec_booster --job $CI_NODE_INDEX/$CI_NODE_TOTAL -``` - -WARNING: -Test Boosters reports usage statistics to the author. +**Related topics**: -You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec -job split into three separate jobs. +- [Parallelize large jobs](../jobs/job_control.md#parallelize-large-jobs). -#### Parallel `matrix` jobs +#### `parallel:matrix` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15356) in GitLab 13.3. +> - The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452). -Use `matrix:` to run a job multiple times in parallel in a single pipeline, +Use `parallel:matrix` to run a job multiple times in parallel in a single pipeline, but with different variable values for each instance of the job. -There can be from 2 to 50 jobs. -Jobs can only run in parallel if there are multiple runners, or a single runner is -configured to run multiple jobs concurrently. +Multiple runners must exist, or a single runner must be configured to run multiple jobs concurrently. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: A numeric value from `2` to `50`. -Every job gets the same `CI_NODE_TOTAL` [CI/CD variable](../variables/index.md#predefined-cicd-variables) value, and a unique `CI_NODE_INDEX` value. +**Example of `parallel:matrix`**: ```yaml deploystacks: @@ -3532,7 +3456,7 @@ deploystacks: STACK: [data, processing] ``` -The following example generates 10 parallel `deploystacks` jobs, each with different values +The example generates 10 parallel `deploystacks` jobs, each with different values for `PROVIDER` and `STACK`: ```plaintext @@ -3548,105 +3472,29 @@ deploystacks: [vultr, data] deploystacks: [vultr, processing] ``` -The job naming style was [improved in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/issues/230452). - -##### One-dimensional `matrix` jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/26362) in GitLab 13.5. - -You can also have one-dimensional matrices with a single job: - -```yaml -deploystacks: - stage: deploy - script: - - bin/deploy - parallel: - matrix: - - PROVIDER: [aws, ovh, gcp, vultr] -``` - -##### Parallel `matrix` trigger jobs - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/270957) in GitLab 13.10. - -Use `matrix:` to run a [trigger](#trigger) job multiple times in parallel in a single pipeline, -but with different variable values for each instance of the job. - -```yaml -deploystacks: - stage: deploy - trigger: - include: path/to/child-pipeline.yml - parallel: - matrix: - - PROVIDER: aws - STACK: [monitoring, app1] - - PROVIDER: ovh - STACK: [monitoring, backup] - - PROVIDER: [gcp, vultr] - STACK: [data] -``` - -This example generates 6 parallel `deploystacks` trigger jobs, each with different values -for `PROVIDER` and `STACK`, and they create 6 different child pipelines with those variables. - -```plaintext -deploystacks: [aws, monitoring] -deploystacks: [aws, app1] -deploystacks: [ovh, monitoring] -deploystacks: [ovh, backup] -deploystacks: [gcp, data] -deploystacks: [vultr, data] -``` +**Related topics**: -In [GitLab 14.1 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/239737), you can -use the variables defined in `parallel: matrix` with the [`tags`](#tags) keyword for -dynamic runner selection. - -```yaml -deploystacks: - stage: deploy - parallel: - matrix: - - PROVIDER: aws - STACK: [monitoring, app1] - - PROVIDER: gcp - STACK: [data] - tags: - - ${PROVIDER}-${STACK} -``` +- [Run a one-dimensional matrix of parallel jobs](../jobs/job_control.md#run-a-one-dimensional-matrix-of-parallel-jobs). +- [Run a matrix of triggered parallel jobs](../jobs/job_control.md#run-a-matrix-of-parallel-trigger-jobs). ### `trigger` > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8997) in GitLab Premium 11.8. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/199224) to GitLab Free in 12.8. -Use `trigger` to define a downstream pipeline trigger. When GitLab starts a `trigger` job, -a downstream pipeline is created. - -Jobs with `trigger` can only use a [limited set of keywords](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). -For example, you can't run commands with [`script`](#script), [`before_script`](#before_script), -or [`after_script`](#after_script). - -You can use this keyword to create two different types of downstream pipelines: +Use `trigger` to start a downstream pipeline that is either: -- [Multi-project pipelines](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file) -- [Child pipelines](../pipelines/parent_child_pipelines.md) +- [A multi-project pipeline](../pipelines/multi_project_pipelines.md). +- [A child pipeline](../pipelines/parent_child_pipelines.md). -In [GitLab 13.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/197140/), you can -view which job triggered a downstream pipeline. In the [pipeline graph](../pipelines/index.md#visualize-pipelines), -hover over the downstream pipeline job. +**Keyword type**: Job keyword. You can use it only as part of a job. -In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you -can use [`when:manual`](#when) in the same job as `trigger`. In GitLab 13.4 and -earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. -You [cannot start `manual` trigger jobs with the API](https://gitlab.com/gitlab-org/gitlab/-/issues/284086). +**Possible inputs**: -#### Basic `trigger` syntax for multi-project pipelines +- For multi-project pipelines, path to the downstream project. +- For child pipelines, path to the child pipeline CI/CD configuration file. -You can configure a downstream trigger by using the `trigger` keyword -with a full path to a downstream project: +**Example of `trigger` for multi-project pipeline**: ```yaml rspec: @@ -3658,47 +3506,7 @@ staging: trigger: my/deployment ``` -#### Complex `trigger` syntax for multi-project pipelines - -You can configure a branch name that GitLab uses to create -a downstream pipeline with: - -```yaml -rspec: - stage: test - script: bundle exec rspec - -staging: - stage: deploy - trigger: - project: my/deployment - branch: stable -``` - -To mirror the status from a triggered pipeline: - -```yaml -trigger_job: - trigger: - project: my/project - strategy: depend -``` - -To mirror the status from an upstream pipeline: - -```yaml -upstream_bridge: - stage: test - needs: - pipeline: other/project -``` - -#### `trigger` syntax for child pipeline - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/16094) in GitLab 12.7. - -To create a [child pipeline](../pipelines/parent_child_pipelines.md), specify the path to the -YAML file that contains the configuration of the child pipeline: +**Example of `trigger` for child pipelines**: ```yaml trigger_job: @@ -3706,71 +3514,36 @@ trigger_job: include: path/to/child-pipeline.yml ``` -Similar to [multi-project pipelines](../pipelines/multi_project_pipelines.md#mirror-status-of-a-triggered-pipeline-in-the-trigger-job), -it's possible to mirror the status from a triggered pipeline: - -```yaml -trigger_job: - trigger: - include: - - local: path/to/child-pipeline.yml - strategy: depend -``` - -##### Trigger child pipeline with generated configuration file - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35632) in GitLab 12.9. - -You can also trigger a child pipeline from a [dynamically generated configuration file](../pipelines/parent_child_pipelines.md#dynamic-child-pipelines): - -```yaml -generate-config: - stage: build - script: generate-ci-config > generated-config.yml - artifacts: - paths: - - generated-config.yml - -child-pipeline: - stage: test - trigger: - include: - - artifact: generated-config.yml - job: generate-config -``` - -The `generated-config.yml` is extracted from the artifacts and used as the configuration -for triggering the child pipeline. +**Additional details**: -##### Trigger child pipeline with files from another project +- Jobs with `trigger` can only use a [limited set of keywords](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). + For example, you can't run commands with [`script`](#script), [`before_script`](#before_script), + or [`after_script`](#after_script). +- In [GitLab 13.5 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/201938), you + can use [`when:manual`](#when) in the same job as `trigger`. In GitLab 13.4 and + earlier, using them together causes the error `jobs:#{job-name} when should be on_success, on_failure or always`. +- In [GitLab 13.2 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/197140/), you can + view which job triggered a downstream pipeline in the [pipeline graph](../pipelines/index.md#visualize-pipelines). -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/205157) in GitLab 13.5. +**Related topics**: -To trigger child pipelines with files from another private project under the same -GitLab instance, use [`include:file`](#includefile): +- [Multi-project pipeline configuration examples](../pipelines/multi_project_pipelines.md#define-multi-project-pipelines-in-your-gitlab-ciyml-file). +- [Child pipeline configuration examples](../pipelines/parent_child_pipelines.md#examples). +- To force a rebuild of a specific branch, tag, or commit, you can + [use an API call with a trigger token](../triggers/index.md). + The trigger token is different than the `trigger` keyword. -```yaml -child-pipeline: - trigger: - include: - - project: 'my-group/my-pipeline-library' - ref: 'main' - file: '/path/to/child-pipeline.yml' -``` +#### `trigger:strategy` -#### Linking pipelines with `trigger:strategy` +Use `trigger:strategy` to force the `trigger` job to wait for the downstream pipeline to complete +before it is marked as **success**. -By default, the `trigger` job completes with the `success` status -as soon as the downstream pipeline is created. +This behavior is different than the default, which is for the `trigger` job to be marked as +**success** as soon as the downstream pipeline is created. -To force the `trigger` job to wait for the downstream (multi-project or child) pipeline to complete, use -`strategy: depend`. This setting makes the trigger job wait with a "running" status until the triggered -pipeline completes. At that point, the `trigger` job completes and displays the same status as -the downstream job. +This setting makes your pipeline execution linear rather than parallel. -This setting can help keep your pipeline execution linear. In the following example, jobs from -subsequent stages wait for the triggered pipeline to successfully complete before -starting, which reduces parallelization. +**Example of `trigger:strategy`**: ```yaml trigger_job: @@ -3779,14 +3552,8 @@ trigger_job: strategy: depend ``` -#### Trigger a pipeline by API call - -To force a rebuild of a specific branch, tag, or commit, you can use an API call -with a trigger token. - -The trigger token is different than the [`trigger`](#trigger) keyword. - -[Read more in the triggers documentation.](../triggers/index.md) +In this example, jobs from subsequent stages wait for the triggered pipeline to +successfully complete before starting. ### `interruptible` @@ -3800,7 +3567,8 @@ a new pipeline starts on the same branch. You can't cancel subsequent jobs after a job with `interruptible: false` starts. -**Keyword type**: Job keyword. You can use it only as part of a job. +**Keyword type**: Job keyword. You can use it only as part of a job or in the +[`default:` section](#default). **Possible inputs**: `true` or `false` (default). @@ -3846,304 +3614,180 @@ In this example, a new pipeline causes a running pipeline to be: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15536) in GitLab 12.7. -Sometimes running multiple jobs at the same time in an environment -can lead to errors during the deployment. - -To avoid these errors, use the `resource_group` attribute to make sure that -the runner doesn't run certain jobs concurrently. Resource groups behave similar -to semaphores in other programming languages. - -When the `resource_group` keyword is defined for a job in the `.gitlab-ci.yml` file, -job executions are mutually exclusive across different pipelines for the same project. -If multiple jobs belonging to the same resource group are enqueued simultaneously, -only one of the jobs is picked by the runner. The other jobs wait until the -`resource_group` is free. +Use `resource_group` to create a [resource group](../resource_groups/index.md) that +ensures a job is mutually exclusive across different pipelines for the same project. -For example: - -```yaml -deploy-to-production: - script: deploy - resource_group: production -``` +For example, if multiple jobs that belong to the same resource group are queued simultaneously, +only one of the jobs starts. The other jobs wait until the `resource_group` is free. -In this case, two `deploy-to-production` jobs in two separate pipelines can never run at the same time. As a result, -you can ensure that concurrent deployments never happen to the production environment. +Resource groups behave similar to semaphores in other programming languages. You can define multiple resource groups per environment. For example, -when deploying to physical devices, you may have multiple physical devices. Each device -can be deployed to, but there can be only one deployment per device at any given time. +when deploying to physical devices, you might have multiple physical devices. Each device +can be deployed to, but only one deployment can occur per device at any given time. -The `resource_group` value can only contain letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. -It can't start or end with `/`. - -For more information, see [Resource Group documentation](../resource_groups/index.md). - -#### Pipeline-level concurrency control with Cross-Project/Parent-Child pipelines - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39057) in GitLab 13.9. +**Keyword type**: Job keyword. You can use it only as part of a job. -You can define `resource_group` for downstream pipelines that are sensitive to concurrent -executions. The [`trigger` keyword](#trigger) can trigger downstream pipelines. The -[`resource_group` keyword](#resource_group) can co-exist with it. This is useful to control the -concurrency for deployment pipelines, while running non-sensitive jobs concurrently. +**Possible inputs**: Only letters, digits, `-`, `_`, `/`, `$`, `{`, `}`, `.`, and spaces. +It can't start or end with `/`. -The following example has two pipeline configurations in a project. When a pipeline starts running, -non-sensitive jobs are executed first and aren't affected by concurrent executions in other -pipelines. However, GitLab ensures that there are no other deployment pipelines running before -triggering a deployment (child) pipeline. If other deployment pipelines are running, GitLab waits -until those pipelines finish before running another one. +**Example of `resource_group`**: ```yaml -# .gitlab-ci.yml (parent pipeline) - -build: - stage: build - script: echo "Building..." - -test: - stage: test - script: echo "Testing..." - -deploy: - stage: deploy - trigger: - include: deploy.gitlab-ci.yml - strategy: depend - resource_group: AWS-production +deploy-to-production: + script: deploy + resource_group: production ``` -```yaml -# deploy.gitlab-ci.yml (child pipeline) - -stages: - - provision - - deploy - -provision: - stage: provision - script: echo "Provisioning..." +In this example, two `deploy-to-production` jobs in two separate pipelines can never run at the same time. As a result, +you can ensure that concurrent deployments never happen to the production environment. -deployment: - stage: deploy - script: echo "Deploying..." -``` +**Related topics**: -You must define [`strategy: depend`](#linking-pipelines-with-triggerstrategy) -with the `trigger` keyword. This ensures that the lock isn't released until the downstream pipeline -finishes. +- [Pipeline-level concurrency control with cross-project/parent-child pipelines](../resource_groups/index.md#pipeline-level-concurrency-control-with-cross-projectparent-child-pipelines). ### `release` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19298) in GitLab 13.2. Use `release` to create a [release](../../user/project/releases/index.md). -Requires the [`release-cli`](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs) -to be available in your GitLab Runner Docker or shell executor. -These keywords are supported: +The release job must have access to the [`release-cli`](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs), +which must be in the `$PATH`. + +If you use the [Docker executor](https://docs.gitlab.com/runner/executors/docker.html), +you can use this image from the GitLab Container Registry: `registry.gitlab.com/gitlab-org/release-cli:latest` + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: The `release:` subkeys: - [`tag_name`](#releasetag_name) -- [`description`](#releasedescription) - [`name`](#releasename) (optional) +- [`description`](#releasedescription) - [`ref`](#releaseref) (optional) - [`milestones`](#releasemilestones) (optional) - [`released_at`](#releasereleased_at) (optional) - [`assets:links`](#releaseassetslinks) (optional) -The release is created only if the job processes without error. If the Rails API -returns an error during release creation, the `release` job fails. - -#### `release-cli` Docker image - -You must specify the Docker image to use for the `release-cli`: - -```yaml -image: registry.gitlab.com/gitlab-org/release-cli:latest -``` - -#### `release-cli` for shell executors - -> - [Introduced](https://gitlab.com/gitlab-org/release-cli/-/issues/21) in GitLab 13.8. -> - [Changed](https://gitlab.com/gitlab-org/release-cli/-/merge_requests/108): the `release-cli` binaries are also -[available in the Package Registry](https://gitlab.com/jaime/release-cli/-/packages) -starting from GitLab 14.2. - -For GitLab Runner shell executors, you can download and install the `release-cli` manually for your [supported OS and architecture](https://release-cli-downloads.s3.amazonaws.com/latest/index.html). -Once installed, the `release` keyword should be available to you. - -**Install on Unix/Linux** - -1. Download the binary for your system from S3, in the following example for amd64 systems: - - ```shell - curl --location --output /usr/local/bin/release-cli "https://release-cli-downloads.s3.amazonaws.com/latest/release-cli-linux-amd64" - ``` +**Example of `release` keyword**: -Or from the GitLab package registry: - - ```shell - curl --location --output /usr/local/bin/release-cli "https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-cli/packages/generic/release-cli/latest/release-cli-darwin-amd64" - ``` - -1. Give it permissions to execute: - - ```shell - sudo chmod +x /usr/local/bin/release-cli - ``` - -1. Verify `release-cli` is available: - - ```shell - $ release-cli -v - - release-cli version 0.6.0 - ``` - -**Install on Windows PowerShell** - -1. Create a folder somewhere in your system, for example `C:\GitLab\Release-CLI\bin` - - ```shell - New-Item -Path 'C:\GitLab\Release-CLI\bin' -ItemType Directory - ``` - -1. Download the executable file: - - ```shell - PS C:\> Invoke-WebRequest -Uri "https://release-cli-downloads.s3.amazonaws.com/latest/release-cli-windows-amd64.exe" -OutFile "C:\GitLab\Release-CLI\bin\release-cli.exe" - - Directory: C:\GitLab\Release-CLI - Mode LastWriteTime Length Name - ---- ------------- ------ ---- - d----- 3/16/2021 4:17 AM bin - - ``` - -1. Add the directory to your `$env:PATH`: - - ```shell - $env:PATH += ";C:\GitLab\Release-CLI\bin" + ```yaml + release_job: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG # Run this job when a tag is created manually + script: + - echo 'Running the release job.' + release: + name: 'Release $CI_COMMIT_TAG' + description: 'Release created using the release-cli.' ``` -1. Verify `release-cli` is available: - - ```shell - PS C:\> release-cli -v +This example creates a release: - release-cli version 0.6.0 - ``` +- When you push a Git tag. +- When you add a Git tag in the UI at **Repository > Tags**. -#### Use a custom SSL CA certificate authority +**Additional details**: -You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority, -which is used to verify the peer when the `release-cli` creates a release through the API using HTTPS with custom certificates. -The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the -[text representation of the X.509 PEM public-key certificate](https://tools.ietf.org/html/rfc7468#section-5.1) -or the `path/to/file` containing the certificate authority. -For example, to configure this value in the `.gitlab-ci.yml` file, use the following: +- All release jobs, except [trigger](#trigger) jobs, must include the `script` keyword. A release + job can use the output from script commands. If you don't need the script, you can use a placeholder: -```yaml -release: - variables: - ADDITIONAL_CA_CERT_BUNDLE: | - -----BEGIN CERTIFICATE----- - MIIGqTCCBJGgAwIBAgIQI7AVxxVwg2kch4d56XNdDjANBgkqhkiG9w0BAQsFADCB - ... - jWgmPqF3vUbZE0EyScetPJquRFRKIesyJuBFMAs= - -----END CERTIFICATE----- + ```yaml script: - - echo "Create release" - release: - name: 'My awesome release' - tag_name: '$CI_COMMIT_TAG' -``` + - echo 'release job' + ``` -The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a -[custom variable in the UI](../variables/index.md#custom-cicd-variables), -either as a `file`, which requires the path to the certificate, or as a variable, -which requires the text representation of the certificate. + An [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/223856) exists to remove this requirement. -#### `script` +- The `release` section executes after the `script` keyword and before the `after_script`. +- A release is created only if the job's main script succeeds. +- If the release already exists, it is not updated and the job with the `release` keyword fails. +- If you use the [Shell executor](https://docs.gitlab.com/runner/executors/shell.html) or similar, + [install `release-cli`](../../user/project/releases/release_cli.md) on the server where the runner is registered. -All jobs except [trigger](#trigger) jobs must have the `script` keyword. A `release` -job can use the output from script commands, but you can use a placeholder script if -the script is not needed: +**Related topics**: -```yaml -script: - - echo 'release job' -``` +- [CI/CD example of the `release` keyword](../../user/project/releases/index.md#cicd-example-of-the-release-keyword). +- [Create multiple releases in a single pipeline](../../user/project/releases/index.md#create-multiple-releases-in-a-single-pipeline). +- [Use a custom SSL CA certificate authority](../../user/project/releases/index.md#use-a-custom-ssl-ca-certificate-authority). -An [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/223856) exists to remove this requirement in an upcoming version of GitLab. +#### `release:tag_name` -A pipeline can have multiple `release` jobs, for example: +Required. The Git tag for the release. -```yaml -ios-release: - script: - - echo 'iOS release job' - release: - tag_name: v1.0.0-ios - description: 'iOS release v1.0.0' +If the tag does not exist in the project yet, it is created at the same time as the release. +New tags use the SHA associated with the pipeline. -android-release: - script: - - echo 'Android release job' - release: - tag_name: v1.0.0-android - description: 'Android release v1.0.0' -``` +**Keyword type**: Job keyword. You can use it only as part of a job. -#### `release:tag_name` +**Possible inputs**: A tag name. Can use [CI/CD variables](../variables/index.md). -You must specify a `tag_name` for the release. The tag can refer to an existing Git tag or -you can specify a new tag. +**Example of `release:tag_name`**: -When the specified tag doesn't exist in the repository, a new tag is created from the associated SHA of the pipeline. +To create a release when a new tag is added to the project: -For example, when creating a release from a Git tag: +- Use the `$CI_COMMIT_TAG` CI/CD variable as the `tag_name`. +- Use [`rules:if`](#rulesif) or [`only: tags`](#onlyrefs--exceptrefs) to configure + the job to run only for new tags. ```yaml job: + script: echo 'Running the release job for the new tag.' release: tag_name: $CI_COMMIT_TAG description: 'Release description' + rules: + - if: $CI_COMMIT_TAG ``` -It is also possible for the release job to automatically create a new unique tag. In that case, -do not use [`rules`](#rules) or [`only`](#only--except) to configure the job to -only run for tags. - -A semantic versioning example: +To create a release and a new tag at the same time, your [`rules`](#rules) or [`only`](#only--except) +should **not** configure the job to run only for new tags. A semantic versioning example: ```yaml job: + script: echo 'Running the release job and creating a new tag.' release: tag_name: ${MAJOR}_${MINOR}_${REVISION} description: 'Release description' + rules: + - if: $CI_PIPELINE_SOURCE == "schedule" ``` -- The release is created only if the job's main script succeeds. -- If the release already exists, it is not updated and the job with the `release` keyword fails. -- The `release` section executes after the `script` tag and before the `after_script`. - #### `release:name` The release name. If omitted, it is populated with the value of `release: tag_name`. +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: A text string. + +**Example of `release:name`**: + +```yaml + release_job: + stage: release + release: + name: 'Release $CI_COMMIT_TAG' +``` + #### `release:description` -Specifies the long description of the release. You can also specify a file that contains the -description. +The long description of the release. -##### Read description from a file +**Keyword type**: Job keyword. You can use it only as part of a job. -> [Introduced](https://gitlab.com/gitlab-org/release-cli/-/merge_requests/67) in GitLab 13.7. +**Possible inputs**: -You can specify a file in `$CI_PROJECT_DIR` that contains the description. The file must be relative -to the project directory (`$CI_PROJECT_DIR`), and if the file is a symbolic link it can't reside -outside of `$CI_PROJECT_DIR`. The `./path/to/file` and filename can't contain spaces. +- A string with the long description. +- The path to a file that contains the description. Introduced in [GitLab 13.7](https://gitlab.com/gitlab-org/release-cli/-/merge_requests/67). + - The file location must be relative to the project directory (`$CI_PROJECT_DIR`). + - If the file is a symbolic link, it must be in the `$CI_PROJECT_DIR`. + - The `./path/to/file` and filename can't contain spaces. + +**Example of `release:description`**: ```yaml job: @@ -4154,8 +3798,13 @@ job: #### `release:ref` -If the `release: tag_name` doesn't exist yet, the release is created from `ref`. -`ref` can be a commit SHA, another tag name, or a branch name. +The `ref` for the release, if the `release: tag_name` doesn't exist yet. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: + +- A commit SHA, another tag name, or a branch name. #### `release:milestones` @@ -4163,21 +3812,31 @@ The title of each milestone the release is associated with. #### `release:released_at` -The date and time when the release is ready. Defaults to the current date and time if not -defined. Should be enclosed in quotes and expressed in ISO 8601 format. +The date and time when the release is ready. -```json +**Possible inputs**: + +- A date enclosed in quotes and expressed in ISO 8601 format. + +**Example of `release:released_at`**: + +```yaml released_at: '2021-03-15T08:00:00Z' ``` +**Additional details**: + +- If it is not defined, the current date and time is used. + #### `release:assets:links` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/271454) in GitLab 13.12. -Include [asset links](../../user/project/releases/index.md#release-assets) in the release. +Use `release:assets:links` to include [asset links](../../user/project/releases/index.md#release-assets) in the release. -NOTE: -Requires `release-cli` version v0.4.0 or higher. +Requires `release-cli` version v0.4.0 or later. + +**Example of `release:assets:links`**: ```yaml assets: @@ -4190,174 +3849,86 @@ assets: link_type: 'other' # optional ``` -#### Complete example for `release` - -If you combine the previous examples for `release`, you get two options, depending on how you generate the -tags. You can't use these options together, so choose one: - -- To create a release when you push a Git tag, or when you add a Git tag - in the UI by going to **Repository > Tags**: - - ```yaml - release_job: - stage: release - image: registry.gitlab.com/gitlab-org/release-cli:latest - rules: - - if: $CI_COMMIT_TAG # Run this job when a tag is created manually - script: - - echo 'running release_job' - release: - name: 'Release $CI_COMMIT_TAG' - description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION must be defined - tag_name: '$CI_COMMIT_TAG' # elsewhere in the pipeline. - ref: '$CI_COMMIT_TAG' - milestones: - - 'm1' - - 'm2' - - 'm3' - released_at: '2020-07-15T08:00:00Z' # Optional, is auto generated if not defined, or can use a variable. - assets: # Optional, multiple asset links - links: - - name: 'asset1' - url: 'https://example.com/assets/1' - - name: 'asset2' - url: 'https://example.com/assets/2' - filepath: '/pretty/url/1' # optional - link_type: 'other' # optional - ``` - -- To create a release automatically when commits are pushed or merged to the default branch, - using a new Git tag that is defined with variables: - - NOTE: - Environment variables set in `before_script` or `script` are not available for expanding - in the same job. Read more about - [potentially making variables available for expanding](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6400). +### `secrets` **(PREMIUM)** - ```yaml - prepare_job: - stage: prepare # This stage must run before the release stage - rules: - - if: $CI_COMMIT_TAG - when: never # Do not run this job when a tag is created manually - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch - script: - - echo "EXTRA_DESCRIPTION=some message" >> variables.env # Generate the EXTRA_DESCRIPTION and TAG environment variables - - echo "TAG=v$(cat VERSION)" >> variables.env # and append to the variables.env file - artifacts: - reports: - dotenv: variables.env # Use artifacts:reports:dotenv to expose the variables to other jobs - - release_job: - stage: release - image: registry.gitlab.com/gitlab-org/release-cli:latest - needs: - - job: prepare_job - artifacts: true - rules: - - if: $CI_COMMIT_TAG - when: never # Do not run this job when a tag is created manually - - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch - script: - - echo 'running release_job for $TAG' - release: - name: 'Release $TAG' - description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION and the $TAG - tag_name: '$TAG' # variables must be defined elsewhere - ref: '$CI_COMMIT_SHA' # in the pipeline. For example, in the - milestones: # prepare_job - - 'm1' - - 'm2' - - 'm3' - released_at: '2020-07-15T08:00:00Z' # Optional, is auto generated if not defined, or can use a variable. - assets: - links: - - name: 'asset1' - url: 'https://example.com/assets/1' - - name: 'asset2' - url: 'https://example.com/assets/2' - filepath: '/pretty/url/1' # optional - link_type: 'other' # optional - ``` - -#### Release assets as Generic packages - -You can use [Generic packages](../../user/packages/generic_packages/) to host your release assets. -For a complete example, see the [Release assets as Generic packages](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs/examples/release-assets-as-generic-package/) -project. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33014) in GitLab 13.4. -#### `release-cli` command line +Use `secrets` to specify [CI/CD secrets](../secrets/index.md) to: -The entries under the `release` node are transformed into a `bash` command line and sent -to the Docker container, which contains the [release-cli](https://gitlab.com/gitlab-org/release-cli). -You can also call the `release-cli` directly from a `script` entry. +- Retrieve from an external secrets provider. +- Make available in the job as [CI/CD variables](../variables/index.md) + ([`file` type](../variables/index.md#cicd-variable-types) by default). -For example, if you use the YAML described previously: +This keyword must be used with `secrets:vault`. -```shell -release-cli create --name "Release $CI_COMMIT_SHA" --description "Created using the release-cli $EXTRA_DESCRIPTION" --tag-name "v${MAJOR}.${MINOR}.${REVISION}" --ref "$CI_COMMIT_SHA" --released-at "2020-07-15T08:00:00Z" --milestone "m1" --milestone "m2" --milestone "m3" --assets-link "{\"name\":\"asset1\",\"url\":\"https://example.com/assets/1\",\"link_type\":\"other\"} -``` +#### `secrets:vault` -### `secrets` +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in GitLab 13.4 and GitLab Runner 13.4. -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/33014) in GitLab 13.4. +Use `secrets:vault` to specify secrets provided by a [HashiCorp Vault](https://www.vaultproject.io/). -Use `secrets` to specify the [CI/CD Secrets](../secrets/index.md) the job needs. It should be a hash, -and the keys should be the names of the variables that are made available to the job. -The value of each secret is saved in a temporary file. This file's path is stored in these -variables. +**Keyword type**: Job keyword. You can use it only as part of a job. -#### `secrets:vault` **(PREMIUM)** +**Possible inputs**: -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/28321) in GitLab 13.4 and GitLab Runner 13.4. +- `engine:name`: Name of the secrets engine. +- `engine:path`: Path to the secrets engine. +- `path`: Path to the secret. +- `field`: Name of the field where the password is stored. -Use `vault` to specify secrets provided by [Hashicorp's Vault](https://www.vaultproject.io/). +**Example of `secrets:vault`**: -This syntax has multiple forms. The shortest form assumes the use of the -[KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) secrets engine, -mounted at the default path `kv-v2`. The last part of the secret's path is the -field to fetch the value for: +To specify all details explicitly and use the [KV-V2](https://www.vaultproject.io/docs/secrets/kv/kv-v2) secrets engine: ```yaml job: secrets: - DATABASE_PASSWORD: - vault: production/db/password # translates to secret `kv-v2/data/production/db`, field `password` + DATABASE_PASSWORD: # Store the path to the secret in this CI/CD variable + vault: # Translates to secret: `ops/data/production/db`, field: `password` + engine: + name: kv-v2 + path: ops + path: production/db + field: password ``` -You can specify a custom secrets engine path by adding a suffix starting with `@`: +You can shorten this syntax. With the short syntax, `engine:name` and `engine:path` +both default to `kv-v2`: ```yaml job: secrets: - DATABASE_PASSWORD: - vault: production/db/password@ops # translates to secret `ops/data/production/db`, field `password` + DATABASE_PASSWORD: # Store the path to the secret in this CI/CD variable + vault: production/db/password # Translates to secret: `kv-v2/data/production/db`, field: `password` ``` -In the detailed form of the syntax, you can specify all details explicitly: +To specify a custom secrets engine path in the short syntax, add a suffix that starts with `@`: ```yaml job: secrets: - DATABASE_PASSWORD: # translates to secret `ops/data/production/db`, field `password` - vault: - engine: - name: kv-v2 - path: ops - path: production/db - field: password + DATABASE_PASSWORD: # Store the path to the secret in this CI/CD variable + vault: production/db/password@ops # Translates to secret: `ops/data/production/db`, field: `password` ``` -#### `secrets:file` **(PREMIUM)** +#### `secrets:file` > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250695) in GitLab 14.1 and GitLab Runner 14.1. -By default, the secret is passed to the job context as a variable of type -[`file`](../variables/index.md#cicd-variable-types). The value of the -secret is stored in a file and the variable `DATABASE_PASSWORD` contains a path to the file. +Use `secrets:file` to configure the secret to be stored as either a +[`file` or `variable` type CI/CD variable](../variables/index.md#cicd-variable-types) -However, some software does not work with file variables and might require the secret value to be stored -directly in the environment variable. For that case, define a `file` setting: +By default, the secret is passed to the job as a `file` type CI/CD variable. The value +of the secret is stored in the file and the variable contains the path to the file. + +If your software can't use `file` type CI/CD variables, set `file: false` to store +the secret value directly in the variable. + +**Keyword type**: Job keyword. You can use it only as part of a job. + +**Possible inputs**: `true` (default) or `false`. + +**Example of `secrets:file`**: ```yaml job: @@ -4367,11 +3938,10 @@ job: file: false ``` -When you set `file: false`, no files are created for that variable. It contains the secret -itself instead. +**Additional details**: -The `file` is a setting of the secret, so it belongs directly under the variable -name level and not in the `vault` section. +- The `file` keyword is a setting for the CI/CD variable and must be nested under + the CI/CD variable name, not in the `vault` section. ### `pages` @@ -4410,484 +3980,154 @@ You must: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207484) in GitLab 12.9. -Use `inherit:` to control inheritance of globally-defined defaults -and variables. - -To enable or disable the inheritance of all `default:` or `variables:` keywords, use: +Use `inherit:` to [control inheritance of globally-defined defaults and variables](../jobs/index.md#control-the-inheritance-of-default-keywords-and-global-variables). -- `default: true` or `default: false` -- `variables: true` or `variables: false` +#### `inherit:default` -To inherit only a subset of `default:` keywords or `variables:`, specify what -you wish to inherit. Anything not listed is **not** inherited. Use -one of the following formats: +Use `inherit:default` to control the inheritance of [default keywords](#default). -```yaml -inherit: - default: [keyword1, keyword2] - variables: [VARIABLE1, VARIABLE2] -``` - -Or: +**Keyword type**: Job keyword. You can use it only as part of a job. -```yaml -inherit: - default: - - keyword1 - - keyword2 - variables: - - VARIABLE1 - - VARIABLE2 -``` +**Possible inputs**: -In the following example: +- `true` (default) or `false` to enable or disable the inheritance of all default keywords. +- A list of specific default keywords to inherit. -- `rubocop`: - - inherits: Nothing. -- `rspec`: - - inherits: the default `image` and the `WEBHOOK_URL` variable. - - does **not** inherit: the default `before_script` and the `DOMAIN` variable. -- `capybara`: - - inherits: the default `before_script` and `image`. - - does **not** inherit: the `DOMAIN` and `WEBHOOK_URL` variables. -- `karma`: - - inherits: the default `image` and `before_script`, and the `DOMAIN` variable. - - does **not** inherit: `WEBHOOK_URL` variable. +**Example of `inherit:default`:** ```yaml default: - image: 'ruby:2.4' - before_script: - - echo Hello World - -variables: - DOMAIN: example.com - WEBHOOK_URL: https://my-webhook.example.com + retry: 2 + image: ruby:3.0 + interruptible: true -rubocop: +job1: + script: echo "This job does not inherit any default keywords." inherit: default: false - variables: false - script: bundle exec rubocop -rspec: - inherit: - default: [image] - variables: [WEBHOOK_URL] - script: bundle exec rspec - -capybara: - inherit: - variables: false - script: bundle exec capybara - -karma: +job2: + script: echo "This job inherits only the two listed default keywords. It does not inherit 'interruptible'." inherit: - default: true - variables: [DOMAIN] - script: karma + default: + - retry + - image ``` -## `variables` - -> Introduced in GitLab Runner v0.5.0. - -[CI/CD variables](../variables/index.md) are configurable values that are passed to jobs. -They can be set globally and per-job. - -There are two types of variables. - -- [Custom variables](../variables/index.md#custom-cicd-variables): - You can define their values in the `.gitlab-ci.yml` file, in the GitLab UI, - or by using the API. You can also input variables in the GitLab UI when - [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). -- [Predefined variables](../variables/predefined_variables.md): - These values are set by the runner itself. - One example is `CI_COMMIT_REF_NAME`, which is the branch or tag the project is built for. - -After you define a variable, you can use it in all executed commands and scripts. - -Variables are meant for non-sensitive project configuration, for example: - -```yaml -variables: - DEPLOY_SITE: "https://example.com/" - -deploy_job: - stage: deploy - script: - - deploy-script --url $DEPLOY_SITE --path "/" - -deploy_review_job: - stage: deploy - variables: - REVIEW_PATH: "/review" - script: - - deploy-review-script --url $DEPLOY_SITE --path $REVIEW_PATH -``` - -You can use only integers and strings for the variable's name and value. +**Additional details:** -If you define a variable at the top level of the `.gitlab-ci.yml` file, it is global, -meaning it applies to all jobs. If you define a variable in a job, it's available -to that job only. +- You can also list default keywords to inherit on one line: `default: [keyword1, keyword2]` -If a variable of the same name is defined globally and for a specific job, the -[job-specific variable overrides the global variable](../variables/index.md#cicd-variable-precedence). +#### `inherit:variables` -All YAML-defined variables are also set to any linked -[Docker service containers](../services/index.md). +Use `inherit:variables` to control the inheritance of [global variables](#variables) keywords. -You can use [YAML anchors for variables](#yaml-anchors-for-variables). +**Keyword type**: Job keyword. You can use it only as part of a job. -### Prefill variables in manual pipelines +**Possible inputs**: -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. +- `true` (default) or `false` to enable or disable the inheritance of all global variables. +- A list of specific variables to inherit. -Use the `value` and `description` keywords to define [pipeline-level (global) variables that are prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) -when [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually): +**Example of `inherit:variables`:** ```yaml variables: - DEPLOY_ENVIRONMENT: - value: "staging" # Deploy to staging by default - description: "The deployment target. Change this variable to 'canary' or 'production' if needed." -``` - -You cannot set job-level variables to be pre-filled when you run a pipeline manually. - -### Configure runner behavior with variables - -You can use [CI/CD variables](../variables/index.md) to configure how the runner processes Git requests: + VARIABLE1: "This is variable 1" + VARIABLE2: "This is variable 2" + VARIABLE3: "This is variable 3" -- [`GIT_STRATEGY`](../runners/configure_runners.md#git-strategy) -- [`GIT_SUBMODULE_STRATEGY`](../runners/configure_runners.md#git-submodule-strategy) -- [`GIT_CHECKOUT`](../runners/configure_runners.md#git-checkout) -- [`GIT_CLEAN_FLAGS`](../runners/configure_runners.md#git-clean-flags) -- [`GIT_FETCH_EXTRA_FLAGS`](../runners/configure_runners.md#git-fetch-extra-flags) -- [`GIT_DEPTH`](../runners/configure_runners.md#shallow-cloning) (shallow cloning) -- [`GIT_CLONE_PATH`](../runners/configure_runners.md#custom-build-directories) (custom build directories) -- [`TRANSFER_METER_FREQUENCY`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact/cache meter update frequency) -- [`ARTIFACT_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (artifact archiver compression level) -- [`CACHE_COMPRESSION_LEVEL`](../runners/configure_runners.md#artifact-and-cache-settings) (cache archiver compression level) - -You can also use variables to configure how many times a runner -[attempts certain stages of job execution](../runners/configure_runners.md#job-stages-attempts). - -## YAML-specific features - -In your `.gitlab-ci.yml` file, you can use YAML-specific features like anchors (`&`), aliases (`*`), -and map merging (`<<`). Use these features to reduce the complexity -of the code in the `.gitlab-ci.yml` file. - -Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). - -In most cases, the [`extends` keyword](#extends) is more user friendly and you should -use it when possible. - -You can use YAML anchors to merge YAML arrays. - -### Anchors - -YAML has a feature called 'anchors' that you can use to duplicate -content across your document. - -Use anchors to duplicate or inherit properties. Use anchors with [hidden jobs](#hide-jobs) -to provide templates for your jobs. When there are duplicate keys, GitLab -performs a reverse deep merge based on the keys. - -You can't use YAML anchors across multiple files when using the [`include`](#include) -keyword. Anchors are only valid in the file they were defined in. To reuse configuration -from different YAML files, use [`!reference` tags](#reference-tags) or the -[`extends` keyword](#extends). - -The following example uses anchors and map merging. It creates two jobs, -`test1` and `test2`, that inherit the `.job_template` configuration, each -with their own custom `script` defined: - -```yaml -.job_template: &job_configuration # Hidden yaml configuration that defines an anchor named 'job_configuration' - image: ruby:2.6 - services: - - postgres - - redis - -test1: - <<: *job_configuration # Merge the contents of the 'job_configuration' alias - script: - - test1 project +job1: + script: echo "This job does not inherit any global variables." + inherit: + variables: false -test2: - <<: *job_configuration # Merge the contents of the 'job_configuration' alias - script: - - test2 project +job2: + script: echo "This job inherits only the two listed global variables. It does not inherit 'VARIABLE3'." + inherit: + variables: + - VARIABLE1 + - VARIABLE2 ``` -`&` sets up the name of the anchor (`job_configuration`), `<<` means "merge the -given hash into the current one," and `*` includes the named anchor -(`job_configuration` again). The expanded version of this example is: - -```yaml -.job_template: - image: ruby:2.6 - services: - - postgres - - redis +**Additional details:** -test1: - image: ruby:2.6 - services: - - postgres - - redis - script: - - test1 project +- You can also list global variables to inherit on one line: `variables: [VARIABLE1, VARIABLE2]` -test2: - image: ruby:2.6 - services: - - postgres - - redis - script: - - test2 project -``` +## `variables` -You can use anchors to define two sets of services. For example, `test:postgres` -and `test:mysql` share the `script` defined in `.job_template`, but use different -`services`, defined in `.postgres_services` and `.mysql_services`: +[CI/CD variables](../variables/index.md) are configurable values that are passed to jobs. +Use `variables` to create [custom variables](../variables/index.md#custom-cicd-variables). -```yaml -.job_template: &job_configuration - script: - - test project - tags: - - dev +Variables are always available in `script`, `before_script`, and `after_script` commands. +You can also use variables as inputs in some job keywords. -.postgres_services: - services: &postgres_configuration - - postgres - - ruby +**Keyword type**: Global and job keyword. You can use it at the global level, +and also at the job level. -.mysql_services: - services: &mysql_configuration - - mysql - - ruby +If you define `variables` at the global level, each variable is copied to +every job configuration when the pipeline is created. If the job already has that +variable defined, the [job-level variable takes precedence](../variables/index.md#cicd-variable-precedence). -test:postgres: - <<: *job_configuration - services: *postgres_configuration - tags: - - postgres +**Possible inputs**: Variable name and value pairs: -test:mysql: - <<: *job_configuration - services: *mysql_configuration -``` +- The name can use only numbers, letters, and underscores (`_`). +- The value must be a string. -The expanded version is: +**Examples of `variables`:** ```yaml -.job_template: - script: - - test project - tags: - - dev - -.postgres_services: - services: - - postgres - - ruby - -.mysql_services: - services: - - mysql - - ruby - -test:postgres: - script: - - test project - services: - - postgres - - ruby - tags: - - postgres - -test:mysql: - script: - - test project - services: - - mysql - - ruby - tags: - - dev -``` - -You can see that the hidden jobs are conveniently used as templates, and -`tags: [postgres]` overwrites `tags: [dev]`. - -#### YAML anchors for scripts - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5. - -You can use [YAML anchors](#anchors) with [script](#script), [`before_script`](#before_script), -and [`after_script`](#after_script) to use predefined commands in multiple jobs: - -```yaml -.some-script-before: &some-script-before - - echo "Execute this script first" - -.some-script: &some-script - - echo "Execute this script second" - - echo "Execute this script too" - -.some-script-after: &some-script-after - - echo "Execute this script last" - -job1: - before_script: - - *some-script-before - script: - - *some-script - - echo "Execute something, for this job only" - after_script: - - *some-script-after +variables: + DEPLOY_SITE: "https://example.com/" -job2: +deploy_job: + stage: deploy script: - - *some-script-before - - *some-script - - echo "Execute something else, for this job only" - - *some-script-after -``` - -#### YAML anchors for variables - -Use [YAML anchors](#anchors) with `variables` to repeat assignment -of variables across multiple jobs. You can also use YAML anchors when a job -requires a specific `variables` block that would otherwise override the global variables. - -The following example shows how override the `GIT_STRATEGY` variable without affecting -the use of the `SAMPLE_VARIABLE` variable: - -```yaml -# global variables -variables: &global-variables - SAMPLE_VARIABLE: sample_variable_value - ANOTHER_SAMPLE_VARIABLE: another_sample_variable_value + - deploy-script --url $DEPLOY_SITE --path "/" -# a job that must set the GIT_STRATEGY variable, yet depend on global variables -job_no_git_strategy: - stage: cleanup +deploy_review_job: + stage: deploy variables: - <<: *global-variables - GIT_STRATEGY: none - script: echo $SAMPLE_VARIABLE -``` - -### Hide jobs - -If you want to temporarily disable a job, rather than commenting out all the -lines where the job is defined: - -```yaml -# hidden_job: -# script: -# - run test -``` - -Instead, you can start its name with a dot (`.`) and it is not processed by -GitLab CI/CD. In the following example, `.hidden_job` is ignored: - -```yaml -.hidden_job: + REVIEW_PATH: "/review" script: - - run test + - deploy-review-script --url $DEPLOY_SITE --path $REVIEW_PATH ``` -Use this feature to ignore jobs, or use the -[YAML-specific features](#yaml-specific-features) and transform the hidden jobs -into templates. - -### `!reference` tags +**Additional details**: -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/266173) in GitLab 13.9. -> - `rules` keyword support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322992) in GitLab 14.3. +- All YAML-defined variables are also set to any linked [Docker service containers](../services/index.md). +- YAML-defined variables are meant for non-sensitive project configuration. Store sensitive information + in [protected variables](../variables/index.md#protect-a-cicd-variable) or [CI/CD secrets](../secrets/index.md). -Use the `!reference` custom YAML tag to select keyword configuration from other job -sections and reuse it in the current section. Unlike [YAML anchors](#anchors), you can -use `!reference` tags to reuse configuration from [included](#include) configuration -files as well. +**Related topics**: -In the following example, a `script` and an `after_script` from two different locations are -reused in the `test` job: +- You can use [YAML anchors for variables](yaml_optimization.md#yaml-anchors-for-variables). +- [Predefined variables](../variables/predefined_variables.md) are variables the runner + automatically creates and makes available in the job. +- You can [configure runner behavior with variables](../runners/configure_runners.md#configure-runner-behavior-with-variables). -- `setup.yml`: +### `variables:description` - ```yaml - .setup: - script: - - echo creating environment - ``` +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/30101) in GitLab 13.7. -- `.gitlab-ci.yml`: +Use the `description` keyword to define a [pipeline-level (global) variable that is prefilled](../pipelines/index.md#prefill-variables-in-manual-pipelines) +when [running a pipeline manually](../pipelines/index.md#run-a-pipeline-manually). - ```yaml - include: - - local: setup.yml +Must be used with `value`, for the variable value. - .teardown: - after_script: - - echo deleting environment +**Keyword type**: Global keyword. You cannot set job-level variables to be pre-filled when you run a pipeline manually. - test: - script: - - !reference [.setup, script] - - echo running my own command - after_script: - - !reference [.teardown, after_script] - ``` +**Possible inputs**: A string. -In the following example, `test-vars-1` reuses all the variables in `.vars`, while `test-vars-2` -selects a specific variable and reuses it as a new `MY_VAR` variable. +**Example of `variables:description`**: ```yaml -.vars: - variables: - URL: "http://my-url.internal" - IMPORTANT_VAR: "the details" - -test-vars-1: - variables: !reference [.vars, variables] - script: - - printenv - -test-vars-2: - variables: - MY_VAR: !reference [.vars, variables, IMPORTANT_VAR] - script: - - printenv +variables: + DEPLOY_ENVIRONMENT: + value: "staging" + description: "The deployment target. Change this variable to 'canary' or 'production' if needed." ``` -You can't reuse a section that already includes a `!reference` tag. Only one level -of nesting is supported. - -## Skip Pipeline - -To push a commit without triggering a pipeline, add `[ci skip]` or `[skip ci]`, using any -capitalization, to your commit message. - -Alternatively, if you are using Git 2.10 or later, use the `ci.skip` [Git push option](../../user/project/push_options.md#push-options-for-gitlab-cicd). -The `ci.skip` push option does not skip merge request -pipelines. - -## Processing Git pushes - -GitLab creates at most four branch and tag pipelines when -pushing multiple changes in a single `git push` invocation. - -This limitation does not affect any of the updated merge request pipelines. -All updated merge requests have a pipeline created when using -[pipelines for merge requests](../pipelines/merge_request_pipelines.md). - ## Deprecated keywords The following keywords are deprecated. @@ -4910,7 +4150,7 @@ Defining `image`, `services`, `cache`, `before_script`, and `after_script` globally is deprecated. Support could be removed from a future release. -Use [`default:`](#custom-default-keyword-values) instead. For example: +Use [`default:`](#default) instead. For example: ```yaml default: diff --git a/doc/ci/yaml/script.md b/doc/ci/yaml/script.md index 626ede21fa5..c1b283ff10f 100644 --- a/doc/ci/yaml/script.md +++ b/doc/ci/yaml/script.md @@ -2,7 +2,6 @@ stage: Verify group: Pipeline Authoring info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # GitLab CI/CD script syntax **(FREE)** @@ -62,7 +61,7 @@ job: ## Set a default `before_script` or `after_script` for all jobs You can use [`before_script`](index.md#before_script) and [`after_script`](index.md#after_script) -with [`default`](index.md#custom-default-keyword-values): +with [`default`](index.md#default): - Use `before_script` with `default` to define a default array of commands that should run before the `script` commands in all jobs. diff --git a/doc/ci/yaml/workflow.md b/doc/ci/yaml/workflow.md new file mode 100644 index 00000000000..67ca1150553 --- /dev/null +++ b/doc/ci/yaml/workflow.md @@ -0,0 +1,150 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab CI/CD `workflow` keyword + +Use the [`workflow`](index.md#workflow) keyword to control when pipelines are created. + +The `workflow` keyword is evaluated before jobs. For example, if a job is configured to run +for tags, but the workflow prevents tag pipelines, the job never runs. + +## Common `if` clauses for `workflow:rules` + +Some example `if` clauses for `workflow: rules`: + +| Example rules | Details | +|------------------------------------------------------|-----------------------------------------------------------| +| `if: '$CI_PIPELINE_SOURCE == "merge_request_event"'` | Control when merge request pipelines run. | +| `if: '$CI_PIPELINE_SOURCE == "push"'` | Control when both branch pipelines and tag pipelines run. | +| `if: $CI_COMMIT_TAG` | Control when tag pipelines run. | +| `if: $CI_COMMIT_BRANCH` | Control when branch pipelines run. | + +See the [common `if` clauses for `rules`](../jobs/job_control.md#common-if-clauses-for-rules) for more examples. + +## `workflow: rules` examples + +In the following example: + +- Pipelines run for all `push` events (changes to branches and new tags). +- Pipelines for push events with `-draft` in the commit message don't run, because + they are set to `when: never`. +- Pipelines for schedules or merge requests don't run either, because no rules evaluate to true for them. + +```yaml +workflow: + rules: + - if: $CI_COMMIT_MESSAGE =~ /-draft$/ + when: never + - if: '$CI_PIPELINE_SOURCE == "push"' +``` + +This example has strict rules, and pipelines do **not** run in any other case. + +Alternatively, all of the rules can be `when: never`, with a final +`when: always` rule. Pipelines that match the `when: never` rules do not run. +All other pipeline types run. For example: + +```yaml +workflow: + rules: + - if: '$CI_PIPELINE_SOURCE == "schedule"' + when: never + - if: '$CI_PIPELINE_SOURCE == "push"' + when: never + - when: always +``` + +This example prevents pipelines for schedules or `push` (branches and tags) pipelines. +The final `when: always` rule runs all other pipeline types, **including** merge +request pipelines. + +## Switch between branch pipelines and merge request pipelines + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/201845) in GitLab 13.8. + +To make the pipeline switch from branch pipelines to merge request pipelines after +a merge request is created, add a `workflow: rules` section to your `.gitlab-ci.yml` file. + +If you use both pipeline types at the same time, [duplicate pipelines](../jobs/job_control.md#avoid-duplicate-pipelines) +might run at the same time. To prevent duplicate pipelines, use the +[`CI_OPEN_MERGE_REQUESTS` variable](../variables/predefined_variables.md). + +The following example is for a project that runs branch and merge request pipelines only, +but does not run pipelines for any other case. It runs: + +- Branch pipelines when a merge request is not open for the branch. +- Merge request pipelines when a merge request is open for the branch. + +```yaml +workflow: + rules: + - if: '$CI_PIPELINE_SOURCE == "merge_request_event"' + - if: '$CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS' + when: never + - if: '$CI_COMMIT_BRANCH' +``` + +If the pipeline is triggered by: + +- A merge request, run a merge request pipeline. For example, a merge request pipeline + can be triggered by a push to a branch with an associated open merge request. +- A change to a branch, but a merge request is open for that branch, do not run a branch pipeline. +- A change to a branch, but without any open merge requests, run a branch pipeline. + +You can also add a rule to an existing `workflow` section to switch from branch pipelines +to merge request pipelines when a merge request is created. + +Add this rule to the top of the `workflow` section, followed by the other rules that +were already present: + +```yaml +workflow: + rules: + - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS && $CI_PIPELINE_SOURCE == "push" + when: never + - ... # Previously defined workflow rules here +``` + +[Triggered pipelines](../triggers/index.md) that run on a branch have a `$CI_COMMIT_BRANCH` +set and could be blocked by a similar rule. Triggered pipelines have a pipeline source +of `trigger` or `pipeline`, so `&& $CI_PIPELINE_SOURCE == "push"` ensures the rule +does not block triggered pipelines. + +## `workflow:rules` templates + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217732) in GitLab 13.0. + +GitLab provides templates that set up `workflow: rules` +for common scenarios. These templates help prevent duplicate pipelines. + +The [`Branch-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/Branch-Pipelines.gitlab-ci.yml) +makes your pipelines run for branches and tags. + +Branch pipeline status is displayed in merge requests that use the branch +as a source. However, this pipeline type does not support any features offered by +[merge request pipelines](../pipelines/merge_request_pipelines.md), like +[pipelines for merged results](../pipelines/pipelines_for_merged_results.md) +or [merge trains](../pipelines/merge_trains.md). +This template intentionally avoids those features. + +To [include](index.md#include) it: + +```yaml +include: + - template: 'Workflows/Branch-Pipelines.gitlab-ci.yml' +``` + +The [`MergeRequest-Pipelines` template](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Workflows/MergeRequest-Pipelines.gitlab-ci.yml) +makes your pipelines run for the default branch, tags, and +all types of merge request pipelines. Use this template if you use any of the +the [pipelines for merge requests features](../pipelines/merge_request_pipelines.md). + +To [include](index.md#include) it: + +```yaml +include: + - template: 'Workflows/MergeRequest-Pipelines.gitlab-ci.yml' +``` diff --git a/doc/ci/yaml/yaml_optimization.md b/doc/ci/yaml/yaml_optimization.md new file mode 100644 index 00000000000..503ba9bbd80 --- /dev/null +++ b/doc/ci/yaml/yaml_optimization.md @@ -0,0 +1,454 @@ +--- +stage: Verify +group: Pipeline Authoring +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference +--- + +# Optimize GitLab CI/CD configuration files **(FREE)** + +You can reduce complexity and duplicated configuration in your GitLab CI/CD configuration +files by using: + +- YAML-specific features like [anchors (`&`)](#anchors), aliases (`*`), and map merging (`<<`). + Read more about the various [YAML features](https://learnxinyminutes.com/docs/yaml/). +- The [`extends` keyword](#use-extends-to-reuse-configuration-sections), + which is more flexible and readable. We recommend you use `extends` where possible. + +## Anchors + +YAML has a feature called 'anchors' that you can use to duplicate +content across your document. + +Use anchors to duplicate or inherit properties. Use anchors with [hidden jobs](../jobs/index.md#hide-jobs) +to provide templates for your jobs. When there are duplicate keys, GitLab +performs a reverse deep merge based on the keys. + +You can use YAML anchors to merge YAML arrays. + +You can't use YAML anchors across multiple files when using the [`include`](index.md#include) +keyword. Anchors are only valid in the file they were defined in. To reuse configuration +from different YAML files, use [`!reference` tags](#reference-tags) or the +[`extends` keyword](#use-extends-to-reuse-configuration-sections). + +The following example uses anchors and map merging. It creates two jobs, +`test1` and `test2`, that inherit the `.job_template` configuration, each +with their own custom `script` defined: + +```yaml +.job_template: &job_configuration # Hidden yaml configuration that defines an anchor named 'job_configuration' + image: ruby:2.6 + services: + - postgres + - redis + +test1: + <<: *job_configuration # Merge the contents of the 'job_configuration' alias + script: + - test1 project + +test2: + <<: *job_configuration # Merge the contents of the 'job_configuration' alias + script: + - test2 project +``` + +`&` sets up the name of the anchor (`job_configuration`), `<<` means "merge the +given hash into the current one," and `*` includes the named anchor +(`job_configuration` again). The expanded version of this example is: + +```yaml +.job_template: + image: ruby:2.6 + services: + - postgres + - redis + +test1: + image: ruby:2.6 + services: + - postgres + - redis + script: + - test1 project + +test2: + image: ruby:2.6 + services: + - postgres + - redis + script: + - test2 project +``` + +You can use anchors to define two sets of services. For example, `test:postgres` +and `test:mysql` share the `script` defined in `.job_template`, but use different +`services`, defined in `.postgres_services` and `.mysql_services`: + +```yaml +.job_template: &job_configuration + script: + - test project + tags: + - dev + +.postgres_services: + services: &postgres_configuration + - postgres + - ruby + +.mysql_services: + services: &mysql_configuration + - mysql + - ruby + +test:postgres: + <<: *job_configuration + services: *postgres_configuration + tags: + - postgres + +test:mysql: + <<: *job_configuration + services: *mysql_configuration +``` + +The expanded version is: + +```yaml +.job_template: + script: + - test project + tags: + - dev + +.postgres_services: + services: + - postgres + - ruby + +.mysql_services: + services: + - mysql + - ruby + +test:postgres: + script: + - test project + services: + - postgres + - ruby + tags: + - postgres + +test:mysql: + script: + - test project + services: + - mysql + - ruby + tags: + - dev +``` + +You can see that the hidden jobs are conveniently used as templates, and +`tags: [postgres]` overwrites `tags: [dev]`. + +### YAML anchors for scripts + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/23005) in GitLab 12.5. + +You can use [YAML anchors](#anchors) with [script](index.md#script), [`before_script`](index.md#before_script), +and [`after_script`](index.md#after_script) to use predefined commands in multiple jobs: + +```yaml +.some-script-before: &some-script-before + - echo "Execute this script first" + +.some-script: &some-script + - echo "Execute this script second" + - echo "Execute this script too" + +.some-script-after: &some-script-after + - echo "Execute this script last" + +job1: + before_script: + - *some-script-before + script: + - *some-script + - echo "Execute something, for this job only" + after_script: + - *some-script-after + +job2: + script: + - *some-script-before + - *some-script + - echo "Execute something else, for this job only" + - *some-script-after +``` + +### YAML anchors for variables + +Use [YAML anchors](#anchors) with `variables` to repeat assignment +of variables across multiple jobs. You can also use YAML anchors when a job +requires a specific `variables` block that would otherwise override the global variables. + +The following example shows how override the `GIT_STRATEGY` variable without affecting +the use of the `SAMPLE_VARIABLE` variable: + +```yaml +# global variables +variables: &global-variables + SAMPLE_VARIABLE: sample_variable_value + ANOTHER_SAMPLE_VARIABLE: another_sample_variable_value + +# a job that must set the GIT_STRATEGY variable, yet depend on global variables +job_no_git_strategy: + stage: cleanup + variables: + <<: *global-variables + GIT_STRATEGY: none + script: echo $SAMPLE_VARIABLE +``` + +## Use `extends` to reuse configuration sections + +You can use the [`extends` keyword](index.md#extends) to reuse configuration in +multiple jobs. It is similar to [YAML anchors](#anchors), but simpler and you can +[use `extends` with `includes`](#use-extends-and-include-together). + +`extends` supports multi-level inheritance. You should avoid using more than three levels, +but you can use as many as eleven. The following example has two levels of inheritance: + +```yaml +.tests: + rules: + - if: $CI_PIPELINE_SOURCE == "push" + +.rspec: + extends: .tests + script: rake rspec + +rspec 1: + variables: + RSPEC_SUITE: '1' + extends: .rspec + +rspec 2: + variables: + RSPEC_SUITE: '2' + extends: .rspec + +spinach: + extends: .tests + script: rake spinach +``` + +### Exclude a key from `extends` + +To exclude a key from the extended content, you must assign it to `null`, for example: + +```yaml +.base: + script: test + variables: + VAR1: base var 1 + +test1: + extends: .base + variables: + VAR1: test1 var 1 + VAR2: test2 var 2 + +test2: + extends: .base + variables: + VAR2: test2 var 2 + +test3: + extends: .base + variables: {} + +test4: + extends: .base + variables: null +``` + +Merged configuration: + +```yaml +test1: + script: test + variables: + VAR1: test1 var 1 + VAR2: test2 var 2 + +test2: + script: test + variables: + VAR1: base var 1 + VAR2: test2 var 2 + +test3: + script: test + variables: + VAR1: base var 1 + +test4: + script: test + variables: null +``` + +### Use `extends` and `include` together + +To reuse configuration from different configuration files, +combine `extends` and [`include`](index.md#include). + +In the following example, a `script` is defined in the `included.yml` file. +Then, in the `.gitlab-ci.yml` file, `extends` refers +to the contents of the `script`: + +- `included.yml`: + + ```yaml + .template: + script: + - echo Hello! + ``` + +- `.gitlab-ci.yml`: + + ```yaml + include: included.yml + + useTemplate: + image: alpine + extends: .template + ``` + +### Merge details + +You can use `extends` to merge hashes but not arrays. +The algorithm used for merge is "closest scope wins," so +keys from the last member always override anything defined on other +levels. For example: + +```yaml +.only-important: + variables: + URL: "http://my-url.internal" + IMPORTANT_VAR: "the details" + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_COMMIT_BRANCH == "stable" + tags: + - production + script: + - echo "Hello world!" + +.in-docker: + variables: + URL: "http://docker-url.internal" + tags: + - docker + image: alpine + +rspec: + variables: + GITLAB: "is-awesome" + extends: + - .only-important + - .in-docker + script: + - rake rspec +``` + +The result is this `rspec` job: + +```yaml +rspec: + variables: + URL: "http://docker-url.internal" + IMPORTANT_VAR: "the details" + GITLAB: "is-awesome" + rules: + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH + - if: $CI_COMMIT_BRANCH == "stable" + tags: + - docker + image: alpine + script: + - rake rspec +``` + +In this example: + +- The `variables` sections merge, but `URL: "http://docker-url.internal"` overwrites `URL: "http://my-url.internal"`. +- `tags: ['docker']` overwrites `tags: ['production']`. +- `script` does not merge, but `script: ['rake rspec']` overwrites + `script: ['echo "Hello world!"']`. You can use [YAML anchors](yaml_optimization.md#anchors) to merge arrays. + +## `!reference` tags + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/266173) in GitLab 13.9. +> - `rules` keyword support [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322992) in GitLab 14.3. + +Use the `!reference` custom YAML tag to select keyword configuration from other job +sections and reuse it in the current section. Unlike [YAML anchors](#anchors), you can +use `!reference` tags to reuse configuration from [included](index.md#include) configuration +files as well. + +In the following example, a `script` and an `after_script` from two different locations are +reused in the `test` job: + +- `setup.yml`: + + ```yaml + .setup: + script: + - echo creating environment + ``` + +- `.gitlab-ci.yml`: + + ```yaml + include: + - local: setup.yml + + .teardown: + after_script: + - echo deleting environment + + test: + script: + - !reference [.setup, script] + - echo running my own command + after_script: + - !reference [.teardown, after_script] + ``` + +In the following example, `test-vars-1` reuses all the variables in `.vars`, while `test-vars-2` +selects a specific variable and reuses it as a new `MY_VAR` variable. + +```yaml +.vars: + variables: + URL: "http://my-url.internal" + IMPORTANT_VAR: "the details" + +test-vars-1: + variables: !reference [.vars, variables] + script: + - printenv + +test-vars-2: + variables: + MY_VAR: !reference [.vars, variables, IMPORTANT_VAR] + script: + - printenv +``` + +You can't reuse a section that already includes a `!reference` tag. Only one level +of nesting is supported. diff --git a/doc/development/adding_database_indexes.md b/doc/development/adding_database_indexes.md index 9ca08ab1dc2..571d2f353d4 100644 --- a/doc/development/adding_database_indexes.md +++ b/doc/development/adding_database_indexes.md @@ -312,3 +312,16 @@ def down remove_concurrent_index_by_name :ci_builds, INDEX_NAME end ``` + +## Test database index changes locally + +You must test the database index changes locally before creating a merge request. + +### Verify indexes created asynchronously + +Use the asynchronous index helpers on your local environment to test changes for creating an index: + +1. Enable the feature flags by running `Feature.enable(:database_async_index_creation)` and `Feature.enable(:database_reindexing)` in the Rails console. +1. Run `bundle exec rails db:migrate` so that it creates an entry in the `postgres_async_indexes` table. +1. Run `bundle exec rails gitlab:db:reindex` so that the index is created asynchronously. +1. To verify the index, open the PostgreSQL console using the [GDK](https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/postgresql.md) command `gdk psql` and run the command `\d <index_name>` to check that your newly created index exists. diff --git a/doc/development/api_styleguide.md b/doc/development/api_styleguide.md index 73814399d2e..df2f3c337cd 100644 --- a/doc/development/api_styleguide.md +++ b/doc/development/api_styleguide.md @@ -240,7 +240,7 @@ it's own file in the [`validators`](https://gitlab.com/gitlab-org/gitlab/-/blob/ ## Internal API -The [internal API](internal_api.md) is documented for internal use. Please keep it up to date so we know what endpoints +The [internal API](internal_api/index.md) is documented for internal use. Please keep it up to date so we know what endpoints different components are making use of. ## Avoiding N+1 problems diff --git a/doc/development/application_slis/index.md b/doc/development/application_slis/index.md index c1d7ac9fa0c..5bc6fffdb48 100644 --- a/doc/development/application_slis/index.md +++ b/doc/development/application_slis/index.md @@ -43,7 +43,7 @@ example: ```ruby Gitlab::Metrics::Sli.initialize_sli(:received_email, [ { - feature_category: :issue_tracking, + feature_category: :team_planning, email_type: :create_issue }, { @@ -110,21 +110,79 @@ will also be incremented: gitlab_sli:received_email:success_total{ feature_category='service_desk', email_type='service_desk' } ``` +So far, only tracking `apdex` using a success rate is supported. If you +need to track errors this way, please upvote +[this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1395) +and leave a comment so we can prioritize this. + ## Using the SLI in service monitoring and alerts -When the application is emitting metrics for the new SLI, those need -to be consumed in the service catalog to result in alerts, and be -included in the error budget for stage groups and GitLab.com's overall -availability. +When the application is emitting metrics for a new SLI, they need +to be consumed from the [metrics catalog](https://gitlab.com/gitlab-com/runbooks/-/tree/master/metrics-catalog) +to result in alerts, and included in the error budget for stage +groups and GitLab.com's overall availability. + +Start by adding the new SLI to the +[Application-SLI library](https://gitlab.com/gitlab-com/runbooks/-/blob/d109886dfd5170793eeb8de3d69aafd4a9da78f6/metrics-catalog/gitlab-slis/library.libsonnet#L4). +After that, add the following information: + +- `name`: the name of the SLI as defined in code. For example + `received_email`. +- `significantLabels`: an array of Prometheus labels that belong to the + metrics. For example: `["email_type"]`. If the significant labels + for the SLI include `feature_category`, the metrics will also + feed into the + [error budgets for stage groups](../stage_group_dashboards.md#error-budget). +- `featureCategory`: if the SLI applies to a single feature category, + you can specify it statically through this field to feed the SLI + into the error budgets for stage groups. +- `description`: a Markdown string explaining the SLI. It will + be shown on dashboards and alerts. +- `kind`: the kind of indicator. Only `sliDefinition.apdexKind` is supported at the moment. + Reach out in + [this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1395) + if you want to implement an SLI for success or error rates. + +When done, run `make generate` to generate recording rules for +the new SLI. This command creates recordings for all services +emitting these metrics aggregated over `significantLabels`. + +Open up a merge request with these changes and request review from a Scalability +team member. + +When these changes are merged, and the aggregations in +[Thanos](https://thanos.gitlab.net) recorded, query Thanos to see +the success ratio of the new aggregated metrics. For example: + +```prometheus +sum by (environment, stage, type)(gitlab_sli_aggregation:rails_request_apdex:apdex:success:rate_1h) +/ +sum by (environment, stage, type)(gitlab_sli_aggregation:rails_request_apdex:apdex:weight:rate_1h) +``` + +This shows the success ratio, which can guide you to set an +appropriate SLO when adding this SLI to a service. + +Then, add the SLI to the appropriate service +catalog file. For example, the [`web` service](https://gitlab.com/gitlab-com/runbooks/-/blob/2b7be37a006c236bd684a4e6a1fbf4c66158292a/metrics-catalog/services/web.jsonnet#L198): + +```jsonnet +rails_requests: + sliLibrary.get('rails_request_apdex') + .generateServiceLevelIndicator({ job: 'gitlab-rails' }) +``` + +To pass extra selectors and override properties of the SLI, see the +[service monitoring documentation](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/README.md). -This is currently being worked on in [this -project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/573). As -part of [this -issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1307) -we will update the documentation. +SLIs with statically defined feature categories can already receive +alerts about the SLI in specified Slack channels. For more information, read the +[alert routing documentation](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/uncategorized/alert-routing.md). +In [this project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/614) +we are extending this so alerts for SLIs with a `feature_category` +label in the souce metrics can also be routed. -For any question, please don't hesitate to createan issue in [the -Scalability issue -tracker](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues) +For any question, please don't hesitate to create an issue in +[the Scalability issue tracker](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues) or come find us in [#g_scalability](https://gitlab.slack.com/archives/CMMF8TKR9) on Slack. diff --git a/doc/development/application_slis/rails_request_apdex.md b/doc/development/application_slis/rails_request_apdex.md index e1ab5368578..b31c7d8756b 100644 --- a/doc/development/application_slis/rails_request_apdex.md +++ b/doc/development/application_slis/rails_request_apdex.md @@ -4,184 +4,171 @@ group: Scalability info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Rails request apdex SLI +# Rails request Apdex SLI > [Introduced](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525) in GitLab 14.4 NOTE: -This SLI is not yet used in [error budgets for stage -groups](../stage_group_dashboards.md#error-budget) or service -monitoring. This is being worked on in [this -project](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/573). +This SLI is used for service monitoring. But not for [error budgets for stage groups](../stage_group_dashboards.md#error-budget) +by default. You can [opt in](#error-budget-attribution-and-ownership). -The request apdex SLI is [an SLI defined in the application](index.md) -that measures the duration of successful requests as an indicator for +The request Apdex SLI (Service Level Indicator) is [an SLI defined in the application](index.md). +It measures the duration of successful requests as an indicator for application performance. This includes the REST and GraphQL API, and the regular controller endpoints. It consists of these counters: 1. `gitlab_sli:rails_request_apdex:total`: This counter gets incremented for every request that did not result in a response - with a 5xx status code. This means that slow failures don't get - counted twice: The request is already counted in the error-SLI. + with a `5xx` status code. It ensures slow failures are not + counted twice, because the request is already counted in the error SLI. 1. `gitlab_sli:rails_request_apdex:success_total`: This counter gets incremented for every successful request that performed faster than - the [defined target duration depending on the endpoint's - urgency](#adjusting-request-urgency). + the [defined target duration depending on the endpoint's urgency](#adjusting-request-urgency). Both these counters are labeled with: 1. `endpoint_id`: The identification of the Rails Controller or the - Grape-API endpoint + Grape-API endpoint. 1. `feature_category`: The feature category specified for that controller or API endpoint. ## Request Apdex SLO -These counters can be combined into a success ratio, the objective for -this ratio is defined in the service catalog per service: +These counters can be combined into a success ratio. The objective for +this ratio is defined in the service catalog per service. For this SLI to meet SLO, +the ratio recorded must be higher than: -1. [Web: 0.998](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/web.jsonnet#L19) -1. [API: 0.995](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/api.jsonnet#L19) -1. [Git: 0.998](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/git.jsonnet#L22) - -This means that for this SLI to meet SLO, the ratio recorded needs to -be higher than those defined above. +- [Web: 0.998](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/web.jsonnet#L19) +- [API: 0.995](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/api.jsonnet#L19) +- [Git: 0.998](https://gitlab.com/gitlab-com/runbooks/blob/master/metrics-catalog/services/git.jsonnet#L22) For example: for the web-service, we want at least 99.8% of requests to be faster than their target duration. -These are the targets we use for alerting and service montoring. So -durations should be set keeping those into account. So we would not -cause alerts. But the goal would be to set the urgency to a target -that users would be satisfied with. +We use these targets for alerting and service monitoring. Set durations taking +these targets into account, so we don't cause alerts. The goal, however, is to +set the urgency to a target that satisfies our users. -Both successful measurements and unsuccessful ones have an impact on the +Both successful measurements and unsuccessful ones affect the error budget for stage groups. ## Adjusting request urgency Not all endpoints perform the same type of work, so it is possible to -define different urgencies for different endpoints. An endpoint with a -lower urgency can have a longer request duration than endpoints that -are high urgency. - -Long-running requests are more expensive for our -infrastructure: while one request is being served, the thread remains -occupied for the duration of that request. So nothing else can be handled by that -thread. Because of Ruby's Global VM Lock, the thread might keep the +define different urgency levels for different endpoints. An endpoint with a +lower urgency can have a longer request duration than endpoints with high urgency. + +Long-running requests are more expensive for our infrastructure. While serving +one request, the thread remains occupied for the duration of that request. The thread +can handle nothing else. Due to Ruby's Global VM Lock, the thread might keep the lock and stall other requests handled by the same Puma worker -process. The request is in fact a noisy neighbor for other requests -handled by the worker. This is why the upper bound for a target -duration is capped at 5 seconds. +process. The request is, in fact, a noisy neighbor for other requests +handled by the worker. We cap the upper bound for a target duration at 5 seconds +for this reason. ## Decreasing the urgency (setting a higher target duration) -Increasing the urgency on an existing endpoint can be done on -a case-by-case basis. Please take the following into account: +You can decrease the urgency on an existing endpoint on +a case-by-case basis. Take the following into account: -1. Apdex is about perceived performance, if a user is actively waiting +1. Apdex is about perceived performance. If a user is actively waiting for the result of a request, waiting 5 seconds might not be - acceptable. While if the endpoint is used by an automation - requiring a lot of data, 5 seconds could be okay. + acceptable. However, if the endpoint is used by an automation + requiring a lot of data, 5 seconds could be acceptable. A product manager can help to identify how an endpoint is used. 1. The workload for some endpoints can sometimes differ greatly depending on the parameters specified by the caller. The urgency - needs to accomodate that. In some cases, it might be interesting to + needs to accommodate those differences. In some cases, you could define a separate [application SLI](index.md#defining-a-new-sli) for what the endpoint is doing. When the endpoints in certain cases turn into no-ops, making them very fast, we should ignore these fast requests when setting the target. For example, if the `MergeRequests::DraftsController` is - hit for every merge request being viewed, but doesn't need to - render anything in most cases, then we should pick the target that - would still accomodate the endpoint performing work. + hit for every merge request being viewed, but rarely renders + anything, then we should pick the target that + would still accommodate the endpoint performing work. 1. Consider the dependent resources consumed by the endpoint. If the endpoint - loads a lot of data from Gitaly or the database and this is causing - it to not perform satisfactory. It could be better to optimize the + loads a lot of data from Gitaly or the database, and this causes + unsatisfactory performance, consider optimizing the way the data is loaded rather than increasing the target duration by lowering the urgency. - In cases like this, it might be appropriate to temporarily decrease + In these cases, it might be appropriate to temporarily decrease urgency to make the endpoint meet SLO, if this is bearable for the - infrastructure. In such cases, please link an issue from a code - comment. + infrastructure. In such cases, create a code comment linking to an issue. If the endpoint consumes a lot of CPU time, we should also consider this: these kinds of requests are the kind of noisy neighbors we should try to keep as short as possible. -1. Traffic characteristics should also be taken into account: if the - trafic to the endpoint is bursty, like CI traffic spinning up a +1. Traffic characteristics should also be taken into account. If the + traffic to the endpoint is bursty, like CI traffic spinning up a big batch of jobs hitting the same endpoint, then having these - endpoints take 5s is not acceptable from an infrastructure point of - view. We cannot scale up the fleet fast enough to accomodate for + endpoints take five seconds is unacceptable from an infrastructure point of + view. We cannot scale up the fleet fast enough to accommodate for the incoming slow requests alongside the regular traffic. When lowering the urgency for an existing endpoint, please involve a [Scalability team member](https://about.gitlab.com/handbook/engineering/infrastructure/team/scalability/#team-members) in the review. We can use request rates and durations available in the -logs to come up with a recommendation. Picking a threshold can be done -using the same process as for [increasing -urgency](#increasing-urgency-setting-a-lower-target-duration), picking -a duration that is higher than the SLO for the service. +logs to come up with a recommendation. You can pick a threshold +using the same process as for +[increasing urgency](#increasing-urgency-setting-a-lower-target-duration), +picking a duration that is higher than the SLO for the service. We shouldn't set the longest durations on endpoints in the merge -requests that introduces them, since we don't yet have data to support +requests that introduces them, because we don't yet have data to support the decision. ## Increasing urgency (setting a lower target duration) -When decreasing the target duration, we need to make sure the endpoint +When increasing the urgency, we must make sure the endpoint still meets SLO for the fleet that handles the request. You can use the -information in the logs to determine this: +information in the logs to check: -1. Open [this table in - Kibana](https://log.gprd.gitlab.net/goto/bbb6465c68eb83642269e64a467df3df) +1. Open [this table in Kibana](https://log.gprd.gitlab.net/goto/bbb6465c68eb83642269e64a467df3df) 1. The table loads information for the busiest endpoints by - default. You can speed things up by adding a filter for - `json.caller_id.keyword` and adding the identifier you're intersted - in (for example: `Projects::RawController#show`). + default. To speed the response, add both: + - A filter for `json.caller_id.keyword`. + - The identifier you're interested in, such as `Projects::RawController#show`. 1. Check the [appropriate percentile duration](#request-apdex-slo) for - the service the endpoint is handled by. The overall duration should - be lower than the target you intend to set. + the service handling the endpoint. The overall duration should + be lower than your intended target. -1. If the overall duration is below the intended targed. Please also - check the peaks over time in [this - graph](https://log.gprd.gitlab.net/goto/9319c4a402461d204d13f3a4924a89fc) +1. If the overall duration is below the intended target, check the peaks over time + in [this graph](https://log.gprd.gitlab.net/goto/9319c4a402461d204d13f3a4924a89fc) in Kibana. Here, the percentile in question should not peak above the target duration we want to set. -Since decreasing a threshold too much could result in alerts for the -apdex degradation, please also involve a Scalability team member in -the merge reqeust. +As decreasing a threshold too much could result in alerts for the +Apdex degradation, please also involve a Scalability team member in +the merge request. ## How to adjust the urgency -The urgency can be specified similar to how endpoints [get a feature -category](../feature_categorization/index.md). - -For endpoints that don't have a specific target, the default urgency (1s duration) will be used. +You can specify urgency similar to how endpoints +[get a feature category](../feature_categorization/index.md). Endpoints without a +specific target use the default urgency: 1s duration. These configurations +are available: -The following configurations are available: - -| Urgency | Duration in seconds | Notes | -|----------|---------------------|-----------------------------------------------| -| :high | 0.25s | | -| :medium | 0.5s | | -| :default | 1s | This is the default when nothing is specified | -| :low | 5s | | +| Urgency | Duration in seconds | Notes | +|------------|---------------------|-----------------------------------------------| +| `:high` | 0.25s | | +| `:medium` | 0.5s | | +| `:default` | 1s | The default when nothing is specified. | +| `:low` | 5s | | ### Rails controller -An urgency can be specified for all actions in a controller like this: +An urgency can be specified for all actions in a controller: ```ruby class Boards::ListsController < ApplicationController @@ -189,8 +176,7 @@ class Boards::ListsController < ApplicationController end ``` -To specify the urgency also for certain actions in a controller, they -can be specified like this: +To also specify the urgency for certain actions in a controller: ```ruby class Boards::ListsController < ApplicationController @@ -200,8 +186,7 @@ end ### Grape endpoints -To specify the urgency for an entire API class, this can be done as -follows: +To specify the urgency for an entire API class: ```ruby module API @@ -211,8 +196,7 @@ module API end ``` -To specify the urgency also for certain actions in a API class, they -can be specified like this: +To specify the urgency also for certain actions in a API class: ```ruby module API @@ -232,3 +216,33 @@ get 'client/features', urgency: :low do # endpoint logic end ``` + +### Error budget attribution and ownership + +This SLI is used for service level monitoring. It feeds into the +[error budget for stage +groups](../stage_group_dashboards.md#error-budget). For this +particular SLI, we have opted everyone out by default to give time to +set the correct urgencies on endpoints before it affects a group's +error budget. + +To include this SLI in the error budget, remove the `rails_requests` +from the `ignored_components` array in the entry for your group. Read +more about what is configurable in the +[runbooks documentation](https://gitlab.com/gitlab-com/runbooks/-/tree/master/services#teamsyml). + +For more information, read the epic for +[defining custom SLIs and incorporating them into error budgets](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/525)). +The endpoints for the SLI feed into a group's error budget based on the +[feature category declared on it](../feature_categorization/index.md). + +To know which endpoints are included for your group, you can see the +request rates on the +[group dashboard for your group](https://dashboards.gitlab.net/dashboards/f/stage-groups/stage-groups). +In the **Budget Attribution** row, the **Puma Apdex** log link shows you +how many requests are not meeting a 1s or 5s target. + +Learn more about the content of the dashboard in the documentation for +[Dashboards for stage groups](../stage_group_dashboards.md). For more information +on our exploration of the error budget itself, read the infrastructure issue +[Stage group error budget exploration dashboard](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1365). diff --git a/doc/development/architecture.md b/doc/development/architecture.md index 9accd4a3595..38d0d5d7843 100644 --- a/doc/development/architecture.md +++ b/doc/development/architecture.md @@ -348,7 +348,6 @@ Component statuses are linked to configuration documentation for each component. | [Gitaly](#gitaly) | Git RPC service for handling all Git calls made by GitLab | ✅ | ✅ | ✅ | ✅ | ✅ | ⚙ | ✅ | CE & EE | | [GitLab Exporter](#gitlab-exporter) | Generates a variety of GitLab metrics | ✅ | ✅ | ✅ | ✅ | ✅ | ❌ | ❌ | CE & EE | | [GitLab Geo Node](#gitlab-geo) | Geographically distributed GitLab nodes | ⚙ | ⚙ | ❌ | ❌ | ✅ | ❌ | ⚙ | EE Only | -| [GitLab Kubernetes Agent](#gitlab-kubernetes-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | | [GitLab Pages](#gitlab-pages) | Hosts static websites | ⚙ | ⚙ | ❌ | ❌ | ✅ | ⚙ | ⚙ | CE & EE | | [GitLab Kubernetes Agent](#gitlab-kubernetes-agent) | Integrate Kubernetes clusters in a cloud-native way | ⚙ | ⚙ | ⚙ | ❌ | ❌ | ⤓ | ⚙ | EE Only | | [GitLab self-monitoring: Alertmanager](#alertmanager) | Deduplicates, groups, and routes alerts from Prometheus | ⚙ | ⚙ | ✅ | ⚙ | ✅ | ❌ | ❌ | CE & EE | @@ -977,12 +976,12 @@ in Rails, scheduled to run whenever an SSH key is modified by a user. instead of keys. In this case, `AuthorizedKeysCommand` is replaced with an `AuthorizedPrincipalsCommand`. This extracts a username from the certificate without using the Rails internal API, which is used instead of `key_id` in the -[`/api/internal/allowed`](internal_api.md) call later. +[`/api/internal/allowed`](internal_api/index.md) call later. GitLab Shell also has a few operations that do not involve Gitaly, such as resetting two-factor authentication codes. These are handled in the same way, except there is no round-trip into Gitaly - Rails performs the action as part -of the [internal API](internal_api.md) call, and GitLab Shell streams the +of the [internal API](internal_api/index.md) call, and GitLab Shell streams the response back to the user directly. ## System layout diff --git a/doc/development/avoiding_downtime_in_migrations.md b/doc/development/avoiding_downtime_in_migrations.md index 9418eafa487..a5fc1909551 100644 --- a/doc/development/avoiding_downtime_in_migrations.md +++ b/doc/development/avoiding_downtime_in_migrations.md @@ -377,7 +377,181 @@ ensures that no downtime is needed. This operation does not require downtime. -## Data Migrations +## Migrating `integer` primary keys to `bigint` + +To [prevent the overflow risk](https://gitlab.com/groups/gitlab-org/-/epics/4785) for some tables +with `integer` primary key (PK), we have to migrate their PK to `bigint`. The process to do this +without downtime and causing too much load on the database is described below. + +### Initialize the conversion and start migrating existing data (release N) + +To start the process, add a regular migration to create the new `bigint` columns. Use the provided +`initialize_conversion_of_integer_to_bigint` helper. The helper also creates a database trigger +to keep in sync both columns for any new records ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/migrate/20210608072312_initialize_conversion_of_ci_stages_to_bigint.rb)): + +```ruby +class InitializeConversionOfCiStagesToBigint < ActiveRecord::Migration[6.1] + include Gitlab::Database::MigrationHelpers + + TABLE = :ci_stages + COLUMNS = %i(id) + + def up + initialize_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end + + def down + revert_initialize_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end +end +``` + +Ignore the new `bigint` columns: + +```ruby +module Ci + class Stage < Ci::ApplicationRecord + include IgnorableColumns + ignore_column :id_convert_to_bigint, remove_with: '14.2', remove_after: '2021-08-22' + end +``` + +To migrate existing data, we introduced new type of _batched background migrations_. +Unlike the classic background migrations, built on top of Sidekiq, batched background migrations +don't have to enqueue and schedule all the background jobs at the beginning. +They also have other advantages, like automatic tuning of the batch size, better progress visibility, +and collecting metrics. To start the process, use the provided `backfill_conversion_of_integer_to_bigint` +helper ([example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/migrate/20210608072346_backfill_ci_stages_for_bigint_conversion.rb)): + +```ruby +class BackfillCiStagesForBigintConversion < ActiveRecord::Migration[6.1] + include Gitlab::Database::MigrationHelpers + + TABLE = :ci_stages + COLUMNS = %i(id) + + def up + backfill_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end + + def down + revert_backfill_conversion_of_integer_to_bigint(TABLE, COLUMNS) + end +end +``` + +### Monitor the background migration + +Check how the migration is performing while it's running. Multiple ways to do this are described below. + +#### High-level status of batched background migrations + +See how to [check the status of batched background migrations](../update/index.md#checking-for-background-migrations-before-upgrading). + +#### Query the database + +We can query the related database tables directly. Requires access to read-only replica. +Example queries: + +```sql +-- Get details for batched background migration for given table +SELECT * FROM batched_background_migrations WHERE table_name = 'namespaces'\gx + +-- Get count of batched background migration jobs by status for given table +SELECT + batched_background_migrations.id, batched_background_migration_jobs.status, COUNT(*) +FROM + batched_background_migrations + JOIN batched_background_migration_jobs ON batched_background_migrations.id = batched_background_migration_jobs.batched_background_migration_id +WHERE + table_name = 'namespaces' +GROUP BY + batched_background_migrations.id, batched_background_migration_jobs.status; + +-- Batched background migration progress for given table (based on estimated total number of tuples) +SELECT + m.table_name, + LEAST(100 * sum(j.batch_size) / pg_class.reltuples, 100) AS percentage_complete +FROM + batched_background_migrations m + JOIN batched_background_migration_jobs j ON j.batched_background_migration_id = m.id + JOIN pg_class ON pg_class.relname = m.table_name +WHERE + j.status = 3 AND m.table_name = 'namespaces' +GROUP BY m.id, pg_class.reltuples; +``` + +#### Sidekiq logs + +We can also use the Sidekiq logs to monitor the worker that executes the batched background +migrations: + +1. Sign in to [Kibana](https://log.gprd.gitlab.net) with a `@gitlab.com` email address. +1. Change the index pattern to `pubsub-sidekiq-inf-gprd*`. +1. Add filter for `json.queue: cronjob:database_batched_background_migration`. + +#### PostgerSQL slow queries log + +Slow queries log keeps track of low queries that took above 1 second to execute. To see them +for batched background migration: + +1. Sign in to [Kibana](https://log.gprd.gitlab.net) with a `@gitlab.com` email address. +1. Change the index pattern to `pubsub-postgres-inf-gprd*`. +1. Add filter for `json.endpoint_id.keyword: Database::BatchedBackgroundMigrationWorker`. +1. Optional. To see only updates, add a filter for `json.command_tag.keyword: UPDATE`. +1. Optional. To see only failed statements, add a filter for `json.error_severiry.keyword: ERROR`. +1. Optional. Add a filter by table name. + +#### Grafana dashboards + +To monitor the health of the database, use these additional metrics: + +- [PostgreSQL Tuple Statistics](https://dashboards.gitlab.net/d/000000167/postgresql-tuple-statistics?orgId=1&refresh=1m): if you see high rate of updates for the tables being actively converted, or increasing percentage of dead tuples for this table, it might mean that autovacuum cannot keep up. +- [PostgreSQL Overview](https://dashboards.gitlab.net/d/000000144/postgresql-overview?orgId=1): if you see high system usage or transactions per second (TPS) on the primary database server, it might mean that the migration is causing problems. + +### Prometheus metrics + +Number of [metrics](https://gitlab.com/gitlab-org/gitlab/-/blob/294a92484ce4611f660439aa48eee4dfec2230b5/lib/gitlab/database/background_migration/batched_migration_wrapper.rb#L90-128) +for each batched background migration are published to Prometheus. These metrics can be searched for and +visualized in Thanos ([see an example](https://thanos-query.ops.gitlab.net/graph?g0.expr=sum%20(rate(batched_migration_job_updated_tuples_total%7Benv%3D%22gprd%22%7D%5B5m%5D))%20by%20(migration_id)%20&g0.tab=0&g0.stacked=0&g0.range_input=3d&g0.max_source_resolution=0s&g0.deduplicate=1&g0.partial_response=0&g0.store_matches=%5B%5D&g0.end_input=2021-06-13%2012%3A18%3A24&g0.moment_input=2021-06-13%2012%3A18%3A24)). + +### Swap the columns (release N + 1) + +After the background is completed and the new `bigint` columns are populated for all records, we can +swap the columns. Swapping is done with post-deployment migration. The exact process depends on the +table being converted, but in general it's done in the following steps: + +1. Using the provided `ensure_batched_background_migration_is_finished` helper, make sure the batched +migration has finished ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L13-18)). +If the migration has not completed, the subsequent steps fail anyway. By checking in advance we +aim to have more helpful error message. +1. Create indexes using the `bigint` columns that match the existing indexes using the `integer` +column ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L28-34)). +1. Create foreign keys (FK) using the `bigint` columns that match the existing FKs using the +`integer` column. Do this both for FK referencing other tables, and FKs that reference the table +that is being migrated ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L36-43)). +1. Inside a transaction, swap the columns: + 1. Lock the tables involved. To reduce the chance of hitting a deadlock, we recommended to do this in parent to child order ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L47)). + 1. Rename the columns to swap names ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L49-54)) + 1. Reset the trigger function ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L56-57)). + 1. Swap the defaults ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L59-62)). + 1. Swap the PK constraint (if any) ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L64-68)). + 1. Remove old indexes and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L70-72)). + 1. Remove old FKs (if still present) and rename new ones ([see an example](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb#L74)). + +See example [merge request](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66088), and [migration](https://gitlab.com/gitlab-org/gitlab/-/blob/41fbe34a4725a4e357a83fda66afb382828767b2/db/post_migrate/20210707210916_finalize_ci_stages_bigint_conversion.rb). + +### Remove the trigger and old `integer` columns (release N + 2) + +Using post-deployment migration and the provided `cleanup_conversion_of_integer_to_bigint` helper, +drop the database trigger and the old `integer` columns ([see an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/69714)). + +### Remove ignore rules (release N + 3) + +In the next release after the columns were dropped, remove the ignore rules as we do not need them +anymore ([see an example](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71161)). + +## Data migrations Data migrations can be tricky. The usual approach to migrate data is to take a 3 step approach: diff --git a/doc/development/chaos_endpoints.md b/doc/development/chaos_endpoints.md index b0de00648ba..838235fd795 100644 --- a/doc/development/chaos_endpoints.md +++ b/doc/development/chaos_endpoints.md @@ -37,7 +37,7 @@ For example, when using the [GDK](https://gitlab.com/gitlab-org/gitlab-developme this can be done with the following command: ```shell -GITLAB_CHAOS_SECRET=secret gdk run +GITLAB_CHAOS_SECRET=secret gdk start ``` Replace `secret` with your own secret token. diff --git a/doc/development/cicd/templates.md b/doc/development/cicd/templates.md index b74a1d0d58a..46442aa6106 100644 --- a/doc/development/cicd/templates.md +++ b/doc/development/cicd/templates.md @@ -60,7 +60,7 @@ don't have any other `.gitlab-ci.yml` files. When authoring pipeline templates: - Place any [global keywords](../../ci/yaml/index.md#global-keywords) like `image` - or `before_script` in a [`default`](../../ci/yaml/index.md#custom-default-keyword-values) + or `before_script` in a [`default`](../../ci/yaml/index.md#default) section at the top of the template. - Note clearly in the [code comments](#explain-the-template-with-comments) if the template is designed to be used with the `includes` keyword in an existing @@ -77,7 +77,7 @@ other pipeline configuration. When authoring job templates: -- Do not use [global](../../ci/yaml/index.md#global-keywords) or [`default`](../../ci/yaml/index.md#custom-default-keyword-values) +- Do not use [global](../../ci/yaml/index.md#global-keywords) or [`default`](../../ci/yaml/index.md#default) keywords. When a root `.gitlab-ci.yml` includes a template, global or default keywords might be overridden and cause unexpected behavior. If a job template requires a specific stage, explain in the code comments that users must manually add the stage diff --git a/doc/development/code_review.md b/doc/development/code_review.md index 89516c2168b..7e797309a26 100644 --- a/doc/development/code_review.md +++ b/doc/development/code_review.md @@ -152,7 +152,7 @@ See the [test engineering process](https://about.gitlab.com/handbook/engineering 1. For the code that this change impacts, I believe that the automated tests ([Testing Guide](testing_guide/index.md)) validate functionality that is highly important to users (including consideration of [all test levels](testing_guide/testing_levels.md)). 1. If the existing automated tests do not cover the above functionality, I have added the necessary additional tests or added an issue to describe the automation testing gap and linked it to this MR. 1. I have considered the technical aspects of this change's impact on GitLab.com hosted customers and self-managed customers. -1. I have considered the impact of this change on the frontend, backend, and database portions of the system where appropriate and applied the `~frontend`, `~backend`, and `~database` labels accordingly. +1. I have considered the impact of this change on the frontend, backend, and database portions of the system where appropriate and applied the `~ux`, `~frontend`, `~backend`, and `~database` labels accordingly. 1. I have tested this MR in [all supported browsers](../install/requirements.md#supported-web-browsers), or determined that this testing is not needed. 1. I have confirmed that this change is [backwards compatible across updates](multi_version_compatibility.md), or I have decided that this does not apply. 1. I have properly separated EE content from FOSS, or this MR is FOSS only. diff --git a/doc/development/contributing/issue_workflow.md b/doc/development/contributing/issue_workflow.md index d0f107ba98a..6a7bbb2e302 100644 --- a/doc/development/contributing/issue_workflow.md +++ b/doc/development/contributing/issue_workflow.md @@ -45,7 +45,7 @@ scheduling into milestones. Labeling is a task for everyone. (For some projects, Most issues will have labels for at least one of the following: -- Type. For example: `~feature`, `~bug`, `~tooling`, or `~documentation`. +- Type. For example: `~"type::feature"`, `~"type::bug"`, or `~"type::tooling"`. - Stage. For example: `~"devops::plan"` or `~"devops::create"`. - Group. For example: `~"group::source code"`, `~"group::knowledge"`, or `~"group::editor"`. - Category. For example: `~"Category:Code Analytics"`, `~"Category:DevOps Reports"`, or `~"Category:Templates"`. @@ -70,12 +70,12 @@ issue should have one and only one. The current type labels are: -- `~feature` +- `~"type::feature"` - `~"feature::addition"` - `~"feature::enhancement"` - - `~"feature::maintenance"` -- `~bug` -- `~tooling` +- `~"type::maintenance"` +- `~"type::bug"` +- `~"type::tooling"` - `~"tooling::pipelines"` - `~"tooling::workflow"` - `~"support request"` @@ -168,7 +168,7 @@ their color is `#428BCA`. `<Category Name>` is the category name as it is in the single source of truth for categories at <https://gitlab.com/gitlab-com/www-gitlab-com/blob/master/data/categories.yml>. -For instance, the "DevOps Report" category is represented by the +For instance, the "DevOps Reports" category is represented by the ~"Category:DevOps Reports" label in the `gitlab-org` group since its `devops_reports.name` value is "DevOps Reports". @@ -342,11 +342,11 @@ To create a feature proposal, open an issue on the [issue tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). In order to help track the feature proposals, we have created a -[`feature`](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=feature) label. +[`~"type::feature"`](https://gitlab.com/gitlab-org/gitlab/-/issues?label_name=type::feature) label. For the time being, users that are not members of the project cannot add labels. You can instead ask one of the [core team](https://about.gitlab.com/community/core-team/) -members to add the label `~feature` to the issue or add the following -code snippet right after your description in a new line: `~feature`. +members to add the label `~"type::feature"` to the issue or add the following +code snippet right after your description in a new line: `~"type::feature"`. Please keep feature proposals as small and simple as possible, complex ones might be edited to make them small and simple. diff --git a/doc/development/contributing/merge_request_workflow.md b/doc/development/contributing/merge_request_workflow.md index a521d89db2b..82fd62d8d79 100644 --- a/doc/development/contributing/merge_request_workflow.md +++ b/doc/development/contributing/merge_request_workflow.md @@ -83,6 +83,24 @@ request is as follows: migrations on a fresh database before the MR is reviewed. If the review leads to large changes in the MR, execute the migrations again once the review is complete. 1. Write tests for more complex migrations. +1. If your merge request adds new validations to existing models, to make sure the + data processing is backwards compatible: + + - Ask in the [`#database`](https://gitlab.slack.com/archives/CNZ8E900G) Slack channel + for assistance to execute the database query that checks the existing rows to + ensure existing rows aren't impacted by the change. + - Add the necessary validation with a feature flag to be gradually rolled out + following [the rollout steps](https://about.gitlab.com/handbook/product-development-flow/feature-flag-lifecycle/#rollout). + + If this merge request is urgent, the code owners should make the final call on + whether reviewing existing rows should be included as an immediate follow-up task + to the merge request. + + NOTE: + There isn't a way to know anything about our customers' data on their + [self-managed instances](../../subscriptions/self_managed/index.md), so keep + that in mind for any data implications with your merge request. + 1. Merge requests **must** adhere to the [merge request performance guidelines](../merge_request_performance_guidelines.md). 1. For tests that use Capybara, read [how to write reliable, asynchronous integration tests](https://thoughtbot.com/blog/write-reliable-asynchronous-integration-tests-with-capybara). @@ -240,6 +258,7 @@ requirements. 1. Working and clean code that is commented where needed. 1. [Unit, integration, and system tests](../testing_guide/index.md) that all pass on the CI server. +1. Peer member testing is optional but recommended when the risk of a change is high. This includes when the changes are [far-reaching](https://about.gitlab.com/handbook/engineering/development/#reducing-the-impact-of-far-reaching-work) or are for [components critical for security](../code_review.md#security). 1. Regressions and bugs are covered with tests that reduce the risk of the issue happening again. 1. [Performance guidelines](../merge_request_performance_guidelines.md) have been followed. diff --git a/doc/development/dangerbot.md b/doc/development/dangerbot.md index aca37e2182a..829f6af76be 100644 --- a/doc/development/dangerbot.md +++ b/doc/development/dangerbot.md @@ -118,6 +118,23 @@ However, you can speed these cycles up somewhat by emptying the `.gitlab/ci/rails.gitlab-ci.yml` file in your merge request. Just don't forget to revert the change before merging! +#### Adding labels via Danger + +NOTE: +This is currently applicable to the [`gitlab-org/gitlab`](https://gitlab.com/gitlab-org/gitlab) +project only. + +Danger is often used to improve MR hygiene by adding labels. Instead of calling the +API directly in your `Dangerfile`, add the labels to the `project_helper.labels_to_add` array. +The main `Dangerfile` will then take care of adding the labels to the MR with a single API call. + +#### Shared rules and plugins + +If the rule or plugin you implement can be useful for other projects, think about +upstreaming them to the [`gitlab-org/gitlab-dangerfiles`](https://gitlab.com/gitlab-org/gitlab-dangerfiles) project. + +#### Enable Danger on a project + To enable the Dangerfile on another existing GitLab project, run the following extra steps: diff --git a/doc/development/database/loose_foreign_keys.md b/doc/development/database/loose_foreign_keys.md new file mode 100644 index 00000000000..157c1284512 --- /dev/null +++ b/doc/development/database/loose_foreign_keys.md @@ -0,0 +1,182 @@ +--- +stage: Enablement +group: Database +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Loose foreign keys + +## Problem statement + +In relational databases (including PostgreSQL), foreign keys provide a way to link +two database tables together, and ensure data-consistency between them. In GitLab, +[foreign keys](../foreign_keys.md) are vital part of the database design process. +Most of our database tables have foreign keys. + +With the ongoing database [decomposition work](https://gitlab.com/groups/gitlab-org/-/epics/6168), +linked records might be present on two different database servers. Ensuring data consistency +between two databases is not possible with standard PostgreSQL foreign keys. PostgreSQL +does not support foreign keys operating within a single database server, defining +a link between two database tables in two different database servers over the network. + +Example: + +- Database "Main": `projects` table +- Database "CI": `ci_pipelines` table + +A project can have many pipelines. When a project is deleted, the associated `ci_pipeline` (via the +`project_id` column) records must be also deleted. + +With a multi-database setup, this cannot be achieved with foreign keys. + +## Asynchronous approach + +Our preferred approach to this problem is eventual consistency. With the loose foreign keys +feature, we can configure delayed association cleanup without negatively affecting the +application performance. + +### How it works + +In the previous example, a record in the `projects` table can have multiple `ci_pipeline` +records. To keep the cleanup process separate from the actual parent record deletion, +we can: + +1. Create a `DELETE` trigger on the `projects` table. + Record the deletions in a separate table (`deleted_records`). +1. A job checks the `deleted_records` table every 5 minutes. +1. For each record in the table, delete the associated `ci_pipelines` records + using the `project_id` column. + +NOTE: +For this procedure to work, we must register which tables to clean up asynchronously. + +## Example migration and configuration + +### Configure the model + +First, tell the application that the `projects` table has a new loose foreign key. +You can do this in the `Project` model: + +```ruby +class Project < ApplicationRecord + # ... + + include LooseForeignKey + + loose_foreign_key :ci_pipelines, :project_id, on_delete: :async_delete # or async_nullify + + # ... +end +``` + +This instruction ensures the asynchronous cleanup process knows about the association, and the +how to do the cleanup. In this case, the associated `ci_pipelines` records are deleted. + +### Track record changes + +To know about deletions in the `projects` table, configure a `DELETE` trigger using a database +migration (post-migration). The trigger needs to be configured only once. If the model already has +at least one `loose_foreign_key` definition, then this step can be skipped: + +```ruby +class TrackProjectRecordChanges < Gitlab::Database::Migration[1.0] + include Gitlab::Database::MigrationHelpers::LooseForeignKeyHelpers + + enable_lock_retries! + + def up + track_record_deletions(:projects) + end + + def down + untrack_record_deletions(:projects) + end +end +``` + +### Remove the foreign key + +If there is an existing foreign key, then it can be removed from the database. As of GitLab 14.5, +the following foreign key describes the link between the `projects` and `ci_pipelines` tables: + +```sql +ALTER TABLE ONLY ci_pipelines +ADD CONSTRAINT fk_86635dbd80 +FOREIGN KEY (project_id) +REFERENCES projects(id) +ON DELETE CASCADE; +``` + +The migration should run after the `DELETE` trigger is installed. If the foreign key is deleted +earlier, there is a good chance of introducing data inconsistency which needs manual cleanup: + +```ruby +class RemoveProjectsCiPipelineFk < Gitlab::Database::Migration[1.0] + enable_lock_retries! + + def up + remove_foreign_key_if_exists(:ci_pipelines, :projects, name: "fk_86635dbd80") + end + + def down + add_concurrent_foreign_key(:ci_pipelines, :projects, name: "fk_86635dbd80", column: :project_id, target_column: :id, on_delete: "cascade") + end +end +``` + +At this point, the setup phase is concluded. The deleted `projects` records should be automatically +picked up by the scheduled cleanup worker job. + +## Caveats of loose foreign keys + +### Record creation + +The feature provides an efficient way of cleaning up associated records after the parent record is +deleted. Without foreign keys, it's the application's responsibility to validate if the parent record +exists when a new associated record is created. + +A bad example: record creation with the given ID (`project_id` comes from user input). +In this example, nothing prevents us from passing a random project ID: + +```ruby +Ci::Pipeline.create!(project_id: params[:project_id]) +``` + +A good example: record creation with extra check: + +```ruby +project = Project.find(params[:project_id]) +Ci::Pipeline.create!(project_id: project.id) +``` + +### Association lookup + +Consider the following HTTP request: + +```plaintext +GET /projects/5/pipelines/100 +``` + +The controller action ignores the `project_id` parameter and finds the pipeline using the ID: + +```ruby + def show + # bad, avoid it + pipeline = Ci::Pipeline.find(params[:id]) # 100 +end +``` + +This endpoint still works when the parent `Project` model is deleted. This can be considered a +a data leak which should not happen under normal circumstances: + +```ruby +def show + # good + project = Project.find(params[:project_id]) + pipeline = project.pipelines.find(params[:pipeline_id]) # 100 +end +``` + +NOTE: +This example is unlikely in GitLab, because we usually look up the parent models to perform +permission checks. diff --git a/doc/development/database/multiple_databases.md b/doc/development/database/multiple_databases.md index 0ba752ba3a6..a17ad798305 100644 --- a/doc/development/database/multiple_databases.md +++ b/doc/development/database/multiple_databases.md @@ -88,16 +88,6 @@ test: &test statement_timeout: 120s ``` -### Migrations - -Place any migrations that affect `Ci::CiDatabaseRecord` models -and their tables in two directories: - -- `db/migrate` -- `db/ci_migrate` - -We aim to keep the schema for both tables the same across both databases. - <!-- NOTE: The `validate_cross_joins!` method in `spec/support/database/prevent_cross_joins.rb` references the following heading in the code, so if you make a change to this heading, make sure to update @@ -272,6 +262,62 @@ logic to delete these rows if or whenever necessary in your domain. Finally, this de-normalization and new query also improves performance because it does less joins and needs less filtering. +##### Remove a redundant join + +Sometimes there are cases where a query is doing excess (or redundant) joins. + +A common example occurs where a query is joining from `A` to `C`, via some +table with both foreign keys, `B`. +When you only care about counting how +many rows there are in `C` and if there are foreign keys and `NOT NULL` constraints +on the foreign keys in `B`, then it might be enough to count those rows. +For example, in +[MR 71811](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71811), it was +previously doing `project.runners.count`, which would produce a query like: + +```sql +select count(*) from projects +inner join ci_runner_projects on ci_runner_projects.project_id = projects.id +where ci_runner_projects.runner_id IN (1, 2, 3) +``` + +This was changed to avoid the cross-join by changing the code to +`project.runner_projects.count`. It produces the same response with the +following query: + +```sql +select count(*) from ci_runner_projects +where ci_runner_projects.runner_id IN (1, 2, 3) +``` + +Another common redundant join is joining all the way to another table, +then filtering by primary key when you could have instead filtered on a foreign +key. See an example in +[MR 71614](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71614). The previous +code was `joins(scan: :build).where(ci_builds: { id: build_ids })`, which +generated a query like: + +```sql +select ... +inner join security_scans +inner join ci_builds on security_scans.build_id = ci_builds.id +where ci_builds.id IN (1, 2, 3) +``` + +However, as `security_scans` already has a foreign key `build_id`, the code +can be changed to `joins(:scan).where(security_scans: { build_id: build_ids })`, +which produces the same response with the following query: + +```sql +select ... +inner join security_scans +where security_scans.build_id IN (1, 2, 3) +``` + +Both of these examples of removing redundant joins remove the cross-joins, +but they have the added benefit of producing simpler and faster +queries. + ##### Use `disable_joins` for `has_one` or `has_many` `through:` relations Sometimes a join query is caused by using `has_one ... through:` or `has_many diff --git a/doc/development/database_review.md b/doc/development/database_review.md index dcd5baab177..45be797b720 100644 --- a/doc/development/database_review.md +++ b/doc/development/database_review.md @@ -172,7 +172,9 @@ test its execution using `CREATE INDEX CONCURRENTLY` in the `#database-lab` Slac `gitlab-org/gitlab-foss` (`project_id = 13083`) or the `gitlab-org/gitlab` (`project_id = 278964`) projects provide enough data to serve as a good example. - That means that no query plan should return 0 records or less records than the provided limit (if a limit is included). If a query is used in batching, a proper example batch with adequate included results should be identified and provided. - - If your queries belong to a new feature in GitLab.com and thus they don't return data in production, it's suggested to analyze the query and to provide the plan from a local environment. + - If your queries belong to a new feature in GitLab.com and thus they don't return data in production: + - You may analyze the query and to provide the plan from a local environment. + - `#database-lab` and [postgres.ai](https://postgres.ai/) both allow updates to data (`exec UPDATE issues SET ...`) and creation of new tables and columns (`exec ALTER TABLE issues ADD COLUMN ...`). - More information on how to find the number of actual returned records in [Understanding EXPLAIN plans](understanding_explain_plans.md) - For query changes, it is best to provide both the SQL queries along with the plan _before_ and _after_ the change. This helps spot differences quickly. diff --git a/doc/development/documentation/feature_flags.md b/doc/development/documentation/feature_flags.md index c169af1958e..8fc6f2e2641 100644 --- a/doc/development/documentation/feature_flags.md +++ b/doc/development/documentation/feature_flags.md @@ -71,8 +71,14 @@ Possible version history entries are: > - [Enabled on GitLab.com](issue-link) in GitLab X.X. > - [Enabled on GitLab.com](issue-link) in GitLab X.X. Available to GitLab.com administrators only. > - [Enabled on self-managed](issue-link) in GitLab X.X. -> - [Feature flag <flag name> removed](issue-link) in GitLab X.X. -> - [Generally available](issue-link) in GitLab X.X. +> - [Generally available](issue-link) in GitLab X.Y. [Feature flag <flag name>](issue-link) removed. +``` + +You can combine entries if they happened in the same release: + +```markdown +> - Introduced in GitLab 14.2 [with a flag](../../administration/feature_flags.md) named `ci_include_rules`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/337507) in GitLab 14.3. ``` ## Feature flag documentation examples @@ -84,7 +90,7 @@ The following examples show the progression of a feature flag. FLAG: On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the featured flag](../administration/feature_flags.md) named `forti_token_cloud`. +ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `forti_token_cloud`. The feature is not ready for production use. ``` @@ -105,6 +111,5 @@ And, when the feature is done and fully available to all users: > - Introduced in GitLab 13.7 [with a flag](../../administration/feature_flags.md) named `forti_token_cloud`. Disabled by default. > - [Enabled on self-managed](https://gitlab.com/issue/etc) GitLab 13.8. > - [Enabled on GitLab.com](https://gitlab.com/issue/etc) in GitLab 13.9. -> - [Feature flag removed](https://gitlab.com/issue/etc) in GitLab 14.0. -> - [Generally available](issue-link) in GitLab 14.0. +> - [Generally available](issue-link) in GitLab 14.0. [Feature flag <flag name>](issue-link) removed. ``` diff --git a/doc/development/documentation/index.md b/doc/development/documentation/index.md index 75538fe1fe7..776e5aefd00 100644 --- a/doc/development/documentation/index.md +++ b/doc/development/documentation/index.md @@ -170,22 +170,33 @@ it increases the work of the release managers. ## GitLab `/help` -Every GitLab instance includes the documentation, which is available at `/help` -(`https://gitlab.example.com/help`). For example, <https://gitlab.com/help>. - -The documentation available online on <https://docs.gitlab.com> is deployed every -four hours from the `main` branch of GitLab, Omnibus, and Runner. Therefore, -after a merge request gets merged, it is available online on the same day. -However, it's shipped (and available on `/help`) within the milestone assigned -to the MR. - -For example, let's say your merge request has a milestone set to 11.3, which -a release date of 2018-09-22. If it gets merged on 2018-09-15, it is -available online on 2018-09-15, but, as the feature freeze date has passed, if -the MR does not have a `~"Pick into 11.3"` label, the milestone has to be changed -to 11.4 and it ships with all GitLab packages only on 2018-10-22, -with GitLab 11.4. Meaning, it's available only with `/help` from GitLab -11.4 onward, but available on <https://docs.gitlab.com/> on the same day it was merged. +Every GitLab instance includes documentation at `/help` (`https://gitlab.example.com/help`) +that matches the version of the instance. For example, <https://gitlab.com/help>. + +The documentation available online at <https://docs.gitlab.com> is deployed every +four hours from the default branch of [GitLab, Omnibus, Runner, and Charts](#source-files-and-rendered-web-locations). +After a merge request that updates documentation is merged, it is available online +in 4 hours or less. + +However, it's only available at `/help` on self-managed instances in the next released +version. The date an update is merged can impact which self-managed release the update +is present in. + +For example: + +1. A merge request in `gitlab` updates documentation. It has a milestone of 14.4, + with an expected release date of 2021-10-22. +1. It is merged on 2021-10-19 and available online the same day at <https://docs.gitlab.com>. +1. GitLab 14.4 is released on 2021-10-22, based on the `gitlab` codebase from 2021-10-18 + (one day *before* the update was merged). +1. The change shows up in the 14.5 self-managed release, due to missing the release cutoff + for 14.4. + +The exact cutoff date for each release is flexible, and can be earlier or later +than expected due to holidays, weekends, or other events. In general, MRs merged +by the 17th should be present in the release on the 22nd, though it is not guaranteed. +If it is important that a documentation update is present in that month's release, +merge it as early as possible. ### Linking to `/help` diff --git a/doc/development/documentation/restful_api_styleguide.md b/doc/development/documentation/restful_api_styleguide.md index b3d3e2641b7..03980a42381 100644 --- a/doc/development/documentation/restful_api_styleguide.md +++ b/doc/development/documentation/restful_api_styleguide.md @@ -49,8 +49,8 @@ METHOD /endpoint Supported attributes: -| Attribute | Type | Required | Description | -|:------------|:---------|:---------|:----------------------| +| Attribute | Type | Required | Description | +| :---------- | :------- | :--------------------- | :-------------------- | | `attribute` | datatype | **{check-circle}** Yes | Detailed description. | | `attribute` | datatype | **{dotted-circle}** No | Detailed description. | | `attribute` | datatype | **{dotted-circle}** No | Detailed description. | @@ -80,16 +80,23 @@ to describe the GitLab release that introduced the API call. Use the following table headers to describe the methods. Attributes should always be in code blocks using backticks (`` ` ``). +Sort the attributes in the table: first, required, then alphabetically. + ```markdown -| Attribute | Type | Required | Description | -|:----------|:-----|:---------|:------------| +| Attribute | Type | Required | Description | +| :------------- | :------------ | :--------------------- | :--------------------------------------------------- | +| `user` | string | **{check-circle}** Yes | The GitLab username. | +| `assignee_ids` | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | +| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | ``` Rendered example: -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:--------------------| -| `user` | string | yes | The GitLab username. | +| Attribute | Type | Required | Description | +| :------------- | :------------ | :--------------------- | :--------------------------------------------------- | +| `user` | string | **{check-circle}** Yes | The GitLab username. | +| `assignee_ids` | integer array | **{dotted-circle}** No | The IDs of the users to assign the issue to. | +| `confidential` | boolean | **{dotted-circle}** No | Set an issue to be confidential. Default is `false`. | ## cURL commands @@ -101,12 +108,12 @@ Rendered example: - Prefer to use examples using the personal access token and don't pass data of username and password. -| Methods | Description | -|:------------------------------------------- |:------------------------------------------------------| +| Methods | Description | +| :---------------------------------------------- | :----------------------------------------------------- | | `--header "PRIVATE-TOKEN: <your_access_token>"` | Use this method as is, whenever authentication needed. | -| `--request POST` | Use this method when creating new objects | -| `--request PUT` | Use this method when updating existing objects | -| `--request DELETE` | Use this method when removing existing objects | +| `--request POST` | Use this method when creating new objects | +| `--request PUT` | Use this method when updating existing objects | +| `--request DELETE` | Use this method when removing existing objects | ## cURL Examples diff --git a/doc/development/documentation/site_architecture/deployment_process.md b/doc/development/documentation/site_architecture/deployment_process.md index 1b764ada87b..c038ee96dbf 100644 --- a/doc/development/documentation/site_architecture/deployment_process.md +++ b/doc/development/documentation/site_architecture/deployment_process.md @@ -4,26 +4,158 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Documentation deployment process +# Documentation deployments + +The documentation [release process](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/releases.md) +involves: + +- Merge requests, to make changes to the `main` and relevant stable branches. +- Pipelines, to build and deploy Docker images to the [`gitlab-docs` container registry](https://gitlab.com/gitlab-org/gitlab-docs/container_registry) + for the relevant stable branches. +- Docker images used to build and deploy all the online documentation, including stable versions and the latest documentation. + +Documentation deployments have dependencies on pipelines and Docker images as follows: + +- The latest documentation pipelines and images depend on the stable documentation pipelines and images. +- The Pages deployment pipelines depend on the latest documentation images (which, in turn, depend on the stable + pipelines and images.) + +For general information on using Docker with CI/CD pipelines, see [Docker integration](../../../ci/docker/index.md). + +## Stable branches + +Stable branches for documentation include the relevant stable branches of all the projects required to publish the entire +documentation suite. For example, the stable version of documentation for version `14.4` includes: + +- The [`14.4`](https://gitlab.com/gitlab-org/gitlab-docs/-/tree/14.4) branch of the `gitlab-docs` project. +- The [`14-4-stable-ee`](https://gitlab.com/gitlab-org/gitlab/-/tree/14-4-stable-ee) branch of the `gitlab` project. +- The [`14-4-stable`](https://gitlab.com/gitlab-org/gitlab-runner/-/tree/14-4-stable) branch of the `gitlab-runner` project. +- The [`14-4-stable`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/14-4-stable) branch of the `omnibus-gitlab` project. +- The [`5-4-stable`](https://gitlab.com/gitlab-org/charts/gitlab/-/tree/5-4-stable) branch of the `charts/gitlab` project. + `charts/gitlab` versions are [mapped](https://docs.gitlab.com/charts/installation/version_mappings.html) to GitLab + versions. + +The Technical Writing team +[creates the stable branch](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/releases.md#create-stable-branch-and-docker-image-for-release) +for the `gitlab-docs` project, which makes use of the stable branches created by other teams. + +## Stable documentation + +When merge requests are merged that target stable branches of `gitlab-docs`, a pipeline builds +that stable documentation and deploys it to the registry. For example: + +- [14.4 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/394459635). +- [14.3 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/393774811). +- [14.2 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/393774758). +- [13.12 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/395365202). +- [12.10 merge request pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/395365405). + +In particular, the [`image:docs-single` job](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L655) in each pipeline +takes what is built, and pushes it to the [container registry](https://gitlab.com/gitlab-org/gitlab-docs/container_registry/631635). + +```mermaid +graph TD + A["14.4 MR merged"] + B["14.3 MR merged"] + C["14.2 MR merged"] + D["13.12 MR merged"] + E["12.10 MR merged"] + F{{"Container registry on `gitlab-docs` project"}} + A--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:14.4` image"-->F + B--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:14.3` image"-->F + C--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:14.2` image"-->F + D--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:13.12` image"-->F + E--"`image:docs-single`<br>job runs and pushes<br>`gitlab-docs:12.10` image"-->F +``` -The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/) -contains all needed Dockerfiles to build and deploy <https://docs.gitlab.com>. It -is heavily inspired by Docker's -[Dockerfile](https://github.com/docker/docker.github.io/blob/06ed03db13895bfe867761b6fc2ad40acf6026dd/Dockerfile). +### Rebuild stable documentation images + +To rebuild any of the stable documentation images, create a [new pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/new) +for the stable branch of the image to rebuild. You might do this: + +- To include new documentation changes from an upstream stable branch into a stable version Docker image. For example, + rebuild the `14.4` Docker image to include changes subsequently merged in the `gitlab` project's + [`14-4-stable-ee`](https://gitlab.com/gitlab-org/gitlab/-/tree/14-4-stable-ee) branch. +- To incorporate changes made to the `gitlab-docs` project itself to a stable branch. For example: + - CSS style changes. + - Changes to the [version menu for a new release](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/doc/releases.md#update-dropdown-for-online-versions). + +## Latest documentation + +A Docker image (tagged `latest`) is built that contains: + +- The latest online version of the documentation. +- The documentation from the stable branches of upstream projects. + +The [`image:docs-latest` job](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L678): + +- Pulls the latest documentation from the default branches of the relevant upstream projects. +- Pulls the Docker images previously built by the `image:docs-single` jobs. + +For example, [a pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/399233948) containing the +[`image:docs-latest` job](https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/1733948330): + +```mermaid +graph TD + A["Latest `gitlab`, `gitlab-runner`<br>`omnibus-gitlab`, and `charts`"] + subgraph "Container registry on `gitlab-docs` project" + B["14.4 versioned docs<br>`gitlab-docs:14.4`"] + C["14.3 versioned docs<br>`gitlab-docs:14.3`"] + D["14.2 versioned docs<br>`gitlab-docs:14.2`"] + E["13.12 versioned docs<br>`gitlab-docs:13.12`"] + F["12.10 versioned docs<br>`gitlab-docs:12.10`"] + end + G[["Scheduled pipeline<br>`image:docs-latest` job<br>combines all these"]] + A--"Default branches<br>pulled down"-->G + B--"`gitlab-docs:14.4` image<br>pulled down"-->G + C--"`gitlab-docs:14.3` image<br>pulled down"-->G + D--"`gitlab-docs:14.2` image<br>pulled down"-->G + E--"`gitlab-docs:13.12` image<br>pulled down"-->G + F--"`gitlab-docs:12.10` image<br>pulled down"-->G + H{{"Container registry on gitlab-docs project"}} + G--"Latest `gitlab-docs:latest` image<br>pushed up"-->H +``` + +## Documentation Pages deployment + +[GitLab Docs](https://docs.gitlab.com) is a [Pages site](../../../user/project/pages/index.md) and documentation updates +for it must be deployed to become available. + +The [`pages`](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/4c18963fe0a414ad62f55b9e18f922588b2dd155/.gitlab-ci.yml#L491) +job runs the necessary commands to combine: + +- A very up-to-date build of the `gitlab-docs` site code. +- The latest docs from the default branches of the upstream projects. +- The documentation from `image:docs-latest`. + +For example, [a pipeline](https://gitlab.com/gitlab-org/gitlab-docs/-/pipelines/399233948) containing the +[`pages` job](https://gitlab.com/gitlab-org/gitlab-docs/-/jobs/1733948332). + +```mermaid +graph LR + A{{"Container registry on gitlab-docs project"}} + B[["Scheduled pipeline<br>`pages` and<br>`pages:deploy` job"]] + C([docs.gitlab.com]) + A--"`gitlab-docs:latest`<br>pulled"-->B + B--"Unpacked documentation uploaded"-->C +``` -The following Dockerfiles are used. +## Docker files -| Dockerfile | Docker image | Description | -| ---------- | ------------ | ----------- | -| [`Dockerfile.bootstrap`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.bootstrap) | `gitlab-docs:bootstrap` | Contains all the dependencies that are needed to build the website. If the gems are updated and `Gemfile{,.lock}` changes, the image must be rebuilt. | -| [`Dockerfile.builder.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.builder.onbuild) | `gitlab-docs:builder-onbuild` | Base image to build the docs website. It uses `ONBUILD` to perform all steps and depends on `gitlab-docs:bootstrap`. | -| [`Dockerfile.nginx.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.nginx.onbuild) | `gitlab-docs:nginx-onbuild` | Base image to use for building documentation archives. It uses `ONBUILD` to perform all required steps to copy the archive, and relies upon its parent `Dockerfile.builder.onbuild` that is invoked when building single documentation archives (see the `Dockerfile` of each branch. | -| [`Dockerfile.archives`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.archives) | `gitlab-docs:archives` | Contains all the versions of the website in one archive. It copies all generated HTML files from every version in one location. | +The [`dockerfiles` directory](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/) contains all needed +Dockerfiles to build and deploy <https://docs.gitlab.com>. It is heavily inspired by Docker's +[Dockerfile](https://github.com/docker/docker.github.io/blob/06ed03db13895bfe867761b6fc2ad40acf6026dd/Dockerfile). + +| Dockerfile | Docker image | Description | +|:---------------------------------------------------------------------------------------------------------------------------|:------------------------------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| [`Dockerfile.bootstrap`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.bootstrap) | `gitlab-docs:bootstrap` | Contains all the dependencies that are needed to build the website. If the gems are updated and `Gemfile{,.lock}` changes, the image must be rebuilt. | +| [`Dockerfile.builder.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.builder.onbuild) | `gitlab-docs:builder-onbuild` | Base image to build the docs website. It uses `ONBUILD` to perform all steps and depends on `gitlab-docs:bootstrap`. | +| [`Dockerfile.nginx.onbuild`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.nginx.onbuild) | `gitlab-docs:nginx-onbuild` | Base image to use for building documentation archives. It uses `ONBUILD` to perform all required steps to copy the archive, and relies upon its parent `Dockerfile.builder.onbuild` that is invoked when building single documentation archives (see the `Dockerfile` of each branch) | +| [`Dockerfile.archives`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/dockerfiles/Dockerfile.archives) | `gitlab-docs:archives` | Contains all the versions of the website in one archive. It copies all generated HTML files from every version in one location. | -## How to build the images +### How to build the images -Although build images are built automatically via GitLab CI/CD, you can build -and tag all tooling images locally: +Although build images are built automatically via GitLab CI/CD, you can build and tag all tooling images locally: 1. Make sure you have [Docker installed](https://docs.docker.com/install/). 1. Make sure you're in the `dockerfiles/` directory of the `gitlab-docs` repository. @@ -37,28 +169,3 @@ and tag all tooling images locally: For each image, there's a manual job under the `images` stage in [`.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab-docs/blob/main/.gitlab-ci.yml) which can be invoked at any time. - -## Update an old Docker image with new upstream docs content - -If there are any changes to any of the stable branches of the products that are -not included in the single Docker image, just rerun the pipeline (`https://gitlab.com/gitlab-org/gitlab-docs/pipelines/new`) -for the version in question. - -## Porting new website changes to old versions - -WARNING: -Porting changes to older branches can have unintended effects as we're constantly -changing the backend of the website. Use only when you know what you're doing -and make sure to test locally. - -The website keeps changing and being improved. In order to consolidate -those changes to the stable branches, we'd need to pick certain changes -from time to time. - -If this is not possible or there are many changes, merge main into them: - -```shell -git branch 12.0 -git fetch origin main -git merge origin/main -``` diff --git a/doc/development/documentation/site_architecture/global_nav.md b/doc/development/documentation/site_architecture/global_nav.md index 3845586ce60..6d2b93b9462 100644 --- a/doc/development/documentation/site_architecture/global_nav.md +++ b/doc/development/documentation/site_architecture/global_nav.md @@ -97,20 +97,13 @@ mechanics of what is required is [documented below](#data-file) but, in principl The global nav has five levels: -- **Section** +- Section - Category - Doc - Doc - Doc -The majority of the links available on the nav were added according to the UI. -The match is not perfect, as for some UI nav items the documentation doesn't -apply, and there are also other links to help the new users to discover the -documentation. The docs under **Administration** are ordered alphabetically -for clarity. - -To see the improvements planned, check the -[global nav epic](https://gitlab.com/groups/gitlab-org/-/epics/1599). +You can view this structure in [the navigation.yml file](https://gitlab.com/gitlab-org/gitlab-docs/-/blob/main/content/_data/navigation.yaml). **Do not** [add items](#add-a-navigation-entry) to the global nav without the consent of one of the technical writers. diff --git a/doc/development/documentation/site_architecture/index.md b/doc/development/documentation/site_architecture/index.md index d1736e10000..60894430d15 100644 --- a/doc/development/documentation/site_architecture/index.md +++ b/doc/development/documentation/site_architecture/index.md @@ -124,12 +124,12 @@ Every four hours a scheduled pipeline builds and deploys the docs site. The pipe fetches the current docs from the main project's main branch, builds it with Nanoc and deploys it to <https://docs.gitlab.com>. -If you need to build and deploy the site immediately (must have maintainer level permissions): +To build and deploy the site immediately (must have the Maintainer role): 1. In [`gitlab-docs`](https://gitlab.com/gitlab-org/gitlab-docs), go to **{rocket}** **CI/CD > Schedules**. 1. For the `Build docs.gitlab.com every 4 hours` scheduled pipeline, click the **play** (**{play}**) button. -Read more about the [deployment process](deployment_process.md). +Read more about [documentation deployments](deployment_process.md). ## Using YAML data files @@ -163,6 +163,35 @@ We can then loop over the `versions` array with something like: Note that the data file must have the `yaml` extension (not `yml`) and that we reference the array with a symbol (`:versions`). +## Archived documentation banner + +A banner is displayed on archived documentation pages with the text `This is archived documentation for +GitLab. Go to the latest.` when either: + +- The version of the documentation displayed is not the first version entry in `online` in + `content/_data/versions.yaml`. +- The documentation was built from the default branch (`main`). + +For example, if the `online` entries for `content/_data/versions.yaml` are: + +```yaml +online: + - "14.4" + - "14.3" + - "14.2" +``` + +In this case, the archived documentation banner isn't displayed: + +- For 14.4, the docs built from the `14.4` branch. The branch name is the first entry in `online`. +- For 14.5-pre, the docs built from the default project branch (`main`). + +The archived documentation banner is displayed: + +- For 14.3. +- For 14.2. +- For any other version. + ## Bumping versions of CSS and JavaScript Whenever the custom CSS and JavaScript files under `content/assets/` change, diff --git a/doc/development/documentation/structure.md b/doc/development/documentation/structure.md index 229dbb077fe..fac83af89f4 100644 --- a/doc/development/documentation/structure.md +++ b/doc/development/documentation/structure.md @@ -234,9 +234,21 @@ If you need to add more than one task, consider using subsections for each distinct task. ``` +### Related topics + +If inline links are not sufficient, you can create a topic called **Related topics** +and include a bulleted list of related topics. This topic should be above the Troubleshooting section. + +```markdown +# Related topics + +- [Configure your pipeline](link-to-topic) +- [Trigger a pipeline manually](link-to-topic) +``` + ### Topics and resources pages -This is a page with a list of links that point to important sections +This page has a list of links that point to important sections of documentation for a specific GitLab feature or tool. We do not encourage the use of these types of pages. diff --git a/doc/development/documentation/styleguide/index.md b/doc/development/documentation/styleguide/index.md index 72491ab3a33..1382ec263f2 100644 --- a/doc/development/documentation/styleguide/index.md +++ b/doc/development/documentation/styleguide/index.md @@ -317,8 +317,8 @@ create an issue or an MR to propose a change to the user interface text. #### Feature names -- *Feature names are typically lowercase*. -- *Some features are capitalized*, typically nouns naming GitLab-specific +- Feature names are typically lowercase. +- Some features are capitalized, typically nouns that name GitLab-specific capabilities or tools. See the [word list](word_list.md) for details. @@ -404,13 +404,13 @@ Some contractions, however, should be avoided: | Do | Don't | |------------------------------------------|-----------------------------------------| - | Do *not* install X with Y. | *Don't* install X with Y. | + | Do not install X with Y. | Don't install X with Y. | - Do not use contractions in reference documentation. For example: | Do | Don't | |------------------------------------------|-----------------------------------------| - | Do *not* set a limit greater than 1000. | *Don't* set a limit greater than 1000. | + | Do not set a limit greater than 1000. | Don't set a limit greater than 1000. | | For `parameter1`, the default is 10. | For `parameter1`, the default's 10. | - Avoid contractions in error messages. Examples: @@ -701,7 +701,7 @@ that's best described by a matrix, tables are the best choice. To keep tables accessible and scannable, tables should not have any empty cells. If there is no otherwise meaningful value for a cell, consider entering -*N/A* (for 'not applicable') or *none*. +**N/A** for 'not applicable' or **None**. To help tables be easier to maintain, consider adding additional spaces to the column widths to make them consistent. For example: @@ -1026,18 +1026,13 @@ document to ensure it links to the most recent version of the file. When documenting how to navigate through the GitLab UI: - Always use location, then action. - - `From the **Visibility** list,` (location) `select **Public**.` (action) + - From the **Visibility** dropdown list (location), select **Public** (action). - Be brief and specific. For example: - - Avoid: `Select **Save** for the changes to take effect.` - - Use instead: `Select **Save**.` -- When selecting from high-level UI elements, use the word **on**. - - Avoid: `From the left sidebar...` or `In the left sidebar...` - - Use instead: `On the left sidebar...` -- If a step must include a reason, start the step with it. - - Avoid: `Select the link in the merge request to view the changes.` - - Use instead: `To view the changes, select the link in the merge request.` -- If a step is optional, start the step with the word `Optional` followed by a period. - - `1. Optional. Enter a name for the dog.` + - Do: Select **Save**. + - Do not: Select **Save** for the changes to take effect. +- If a step must include a reason, start the step with it. This helps the user scan more quickly. + - Do: To view the changes, in the merge request, select the link. + - Do not: Select the link in the merge request to view the changes. ### Names for menus @@ -1060,15 +1055,19 @@ Guidance for each individual UI element is in [the word list](word_list.md). To be consistent, use this format when you write navigation steps in a task topic. +```markdown 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **General pipelines**. +``` Another example: +```markdown 1. On the top bar, select **Menu > Groups** and find your group. 1. On the left sidebar, select **Settings > CI/CD**. 1. Expand **General pipelines**. +``` An Admin Area example: @@ -1082,6 +1081,42 @@ To select your avatar: 1. On the top bar, in the top right corner, select your avatar. ``` +### Optional steps + +If a step is optional, start the step with the word `Optional` followed by a period. + +For example: + +```markdown +1. Optional. Enter a description for the job. +``` + +### Documenting multiple fields at once + +If the UI text sufficiently explains the fields in a section, do not include a task step for every field. +Instead, summarize multiple fields in a single task step. + +Use the phrase **Complete the fields**. + +For example: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Push rules**. +1. Complete the fields. + +If you are documenting multiple fields and only one field needs explanation, do it in the same step: + +1. Expand **Push rules**. +1. Complete the fields. **Branch name** must be a regular expression. + +To describe multiple fields, use bullets: + +1. Expand **General pipelines**. +1. Complete the fields. + - **Branch name** must be a regular expression. + - **User** must be a user with at least the **Maintainer** role. + ## Images Images, including screenshots, can help a reader better understand a concept. diff --git a/doc/development/documentation/styleguide/word_list.md b/doc/development/documentation/styleguide/word_list.md index f1e6a147571..595dab09bf5 100644 --- a/doc/development/documentation/styleguide/word_list.md +++ b/doc/development/documentation/styleguide/word_list.md @@ -259,6 +259,16 @@ Use **box** instead of **field** or **text box**. - Do: In the **Variable name** box, enter `my text`. - Do not: In the **Variable name** field, enter `my text`. +However, you can make an exception when you are writing a task and you need to refer to all +of the fields at once. For example: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > CI/CD**. +1. Expand **General pipelines**. +1. Complete the fields. + +Learn more about [documenting multiple fields at once](index.md#documenting-multiple-fields-at-once). + ## foo Do not use **foo** in product documentation. You can use it in our API and contributor documentation, but try to use a clearer and more meaningful example instead. @@ -454,6 +464,13 @@ Do not use **note that** because it's wordy. - Do: You can change the settings. - Do not: Note that you can change the settings. +## on + +When documenting how to select high-level UI elements, use the word **on**. + +- Do: `On the left sidebar...` +- Do not: `From the left sidebar...` or `In the left sidebar...` + ## once The word **once** means **one time**. Don't use it to mean **after** or **when**. @@ -524,8 +541,8 @@ Use lowercase for **runners**. These are the agents that run CI/CD jobs. See als Do not use **(s)** to make a word optionally plural. It can slow down comprehension. For example: -Do: Select the jobs you want. -Do not: Select the job(s) you want. +- Do: Select the jobs you want. +- Do not: Select the job(s) you want. If you can select multiples of something, then write the word as plural. @@ -555,6 +572,10 @@ or collapsing a section, don't include the word **section**. Use **select** with buttons, links, menu items, and lists. **Select** applies to more devices, while **click** is more specific to a mouse. +## Service Desk + +Use title case for **Service Desk**. + ## setup, set up Use **setup** as a noun, and **set up** as a verb. For example: diff --git a/doc/development/documentation/workflow.md b/doc/development/documentation/workflow.md index 90c1137e5c5..782cd3411b1 100644 --- a/doc/development/documentation/workflow.md +++ b/doc/development/documentation/workflow.md @@ -39,7 +39,7 @@ The following are also added by members of the Technical Writing team: - The `~Technical Writing` [team label](../contributing/issue_workflow.md#team-labels). Documentation changes that are not associated with the release of a new or updated feature -do not take the `~feature` label, but still need the `~documentation` label. +do not take the `~"type::feature"` label, but still need the `~documentation` label. They may include: diff --git a/doc/development/elasticsearch.md b/doc/development/elasticsearch.md index 68d8b424331..7c67b3495ba 100644 --- a/doc/development/elasticsearch.md +++ b/doc/development/elasticsearch.md @@ -250,6 +250,11 @@ the migration runs and set it back to that value when the migration is completed will halt the migration if the storage required is not available when the migration runs. The migration must provide the space required in bytes by defining a `space_required_bytes` method. +- `retry_on_failure` - Enable the retry on failure feature. By default, it retries + the migration 30 times. After it runs out of retries, the migration is marked as halted. + To customize the number of retries, pass the `max_attempts` argument: + `retry_on_failure max_attempts: 10` + ```ruby # frozen_string_literal: true @@ -259,6 +264,7 @@ class BatchedMigrationName < Elastic::Migration throttle_delay 10.minutes pause_indexing! space_requirements! + retry_on_failure # ... end @@ -359,17 +365,15 @@ being upgraded to, we do the following: ### Prometheus -GitLab exports [Prometheus -metrics](../administration/monitoring/prometheus/gitlab_metrics.md) relating to -the number of requests and timing for all web/API requests and Sidekiq jobs, +GitLab exports [Prometheus metrics](../administration/monitoring/prometheus/gitlab_metrics.md) +relating to the number of requests and timing for all web/API requests and Sidekiq jobs, which can help diagnose performance trends and compare how Elasticsearch timing is impacting overall performance relative to the time spent doing other things. #### Indexing queues -GitLab also exports [Prometheus -metrics](../administration/monitoring/prometheus/gitlab_metrics.md) for -indexing queues, which can help diagnose performance bottlenecks and determine +GitLab also exports [Prometheus metrics](../administration/monitoring/prometheus/gitlab_metrics.md) +for indexing queues, which can help diagnose performance bottlenecks and determine whether or not your GitLab instance or Elasticsearch server can keep up with the volume of updates. diff --git a/doc/development/experiment_guide/gitlab_experiment.md b/doc/development/experiment_guide/gitlab_experiment.md index 1cd3fefb7cf..af4512dcde0 100644 --- a/doc/development/experiment_guide/gitlab_experiment.md +++ b/doc/development/experiment_guide/gitlab_experiment.md @@ -92,7 +92,7 @@ end ``` When this code executes, the experiment is run, a variant is assigned, and (if within a -controller or view) a `window.gon.experiment.pill_color` object will be available in the +controller or view) a `window.gl.experiments.pill_color` object will be available in the client layer, with details like: - The assigned variant. @@ -522,14 +522,14 @@ shared example: [tracks assignment and records the subject](https://gitlab.com/g This is in flux as of GitLab 13.10, and can't be documented just yet. -Any experiment that's been run in the request lifecycle surfaces in `window.gon.experiment`, +Any experiment that's been run in the request lifecycle surfaces in and `window.gl.experiments`, and matches [this schema](https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/gitlab_experiment/jsonschema/1-0-0) so you can use it when resolving some concepts around experimentation in the client layer. ### Use experiments in Vue With the `gitlab-experiment` component, you can define slots that match the name of the -variants pushed to `window.gon.experiment`. For example, if we alter the `pill_color` +variants pushed to `window.gl.experiments`. For example, if we alter the `pill_color` experiment to just use the default variants of `control` and `candidate` like so: ```ruby @@ -587,7 +587,51 @@ For example, the Vue component for the previously-defined `pill_color` experimen ``` NOTE: -When there is no experiment data in the `window.gon.experiment` object for the given experiment name, the `control` slot will be used, if it exists. +When there is no experiment data in the `window.gl.experiments` object for the given experiment name, the `control` slot will be used, if it exists. + +## Test with Jest + +### Stub Helpers + +You can stub experiments using the `stubExperiments` helper defined in `spec/frontend/__helpers__/experimentation_helper.js`. + +```javascript +import { stubExperiments } from 'helpers/experimentation_helper'; +import { getExperimentData } from '~/experimentation/utils'; + +describe('when my_experiment is enabled', () => { + beforeEach(() => { + stubExperiments({ my_experiment: 'candidate' }); + }); + + it('sets the correct data', () => { + expect(getExperimentData('my_experiment')).toEqual({ experiment: 'my_experiment', variant: 'candidate' }); + }); +}); +``` + +NOTE: +This method of stubbing in Jest specs will not automatically un-stub itself at the end of the test. We merge our stubbed experiment in with all the other global data in `window.gl`. If you need to remove the stubbed experiment(s) after your test or ensure a clean global object before your test, you'll need to manage the global object directly yourself: + +```javascript +desribe('tests that care about global state', () => { + const originalObjects = []; + + beforeEach(() => { + // For backwards compatibility for now, we're using both window.gon & window.gl + originalObjects.push(window.gon, window.gl); + }); + + afterEach(() => { + [window.gon, window.gl] = originalObjects; + }); + + it('stubs experiment in fresh global state', () => { + stubExperiment({ my_experiment: 'candidate' }); + // ... + }); +}) +``` ## Notes on feature flags diff --git a/doc/development/fe_guide/accessibility.md b/doc/development/fe_guide/accessibility.md index c4ebef4c289..e71e414002a 100644 --- a/doc/development/fe_guide/accessibility.md +++ b/doc/development/fe_guide/accessibility.md @@ -23,6 +23,17 @@ they found that "ARIA correlated to higher detectable errors". It is likely that *misuse* of ARIA is a big cause of increased errors, so when in doubt don't use `aria-*`, `role`, and `tabindex` and stick with semantic HTML. +## Enable keyboard navigation on macOS + +By default, macOS limits the <kbd>tab</kbd> key to **Text boxes and lists only**. To enable full keyboard navigation: + +1. Open **System Preferences**. +1. Select **Keyboard**. +1. Open the **Shortcuts** tab. +1. Enable the setting **Use keyboard navigation to move focus between controls**. + +You can read more about enabling browser-specific keyboard navigation on [a11yproject](https://www.a11yproject.com/posts/2017-12-29-macos-browser-keyboard-navigation/). + ## Quick checklist - [Text](#text-inputs-with-accessible-names), diff --git a/doc/development/fe_guide/development_process.md b/doc/development/fe_guide/development_process.md index 4e50621add4..cd82a7dadc3 100644 --- a/doc/development/fe_guide/development_process.md +++ b/doc/development/fe_guide/development_process.md @@ -56,7 +56,7 @@ Please use your best judgment when to use it and please contribute new points th - [ ] **Performance** Have you checked performance? For example do the same thing with 500 comments instead of 1. Document the tests and possible findings in the MR so a reviewer can directly see it. - [ ] Have you tested with a variety of our [supported browsers](../../install/requirements.md#supported-web-browsers)? You can use [browserstack](https://www.browserstack.com/) to be able to access a wide variety of browsers and operating systems. - [ ] Did you check the mobile view? -- [ ] Check the built webpack bundle (For the report run `WEBPACK_REPORT=true gdk run`, then open `webpack-report/index.html`) if we have unnecessary bloat due to wrong references, including libraries multiple times, etc.. If you need help contact the webpack [domain expert](https://about.gitlab.com/handbook/engineering/frontend/#frontend-domain-experts) +- [ ] Check the built webpack bundle (For the report run `WEBPACK_REPORT=true gdk start`, then open `webpack-report/index.html`) if we have unnecessary bloat due to wrong references, including libraries multiple times, etc.. If you need help contact the webpack [domain expert](https://about.gitlab.com/handbook/engineering/frontend/#frontend-domain-experts) - [ ] **Tests** Not only greenfield tests - Test also all bad cases that come to your mind. - [ ] If you have multiple MR's then also smoke test against the final merge. - [ ] Are there any big changes on how and especially how frequently we use the API then let production know about it diff --git a/doc/development/fe_guide/graphql.md b/doc/development/fe_guide/graphql.md index 0229aa0123a..44d43a32803 100644 --- a/doc/development/fe_guide/graphql.md +++ b/doc/development/fe_guide/graphql.md @@ -103,7 +103,6 @@ Default client accepts two parameters: `resolvers` and `config`. - `config` parameter takes an object of configuration settings: - `cacheConfig` field accepts an optional object of settings to [customize Apollo cache](https://www.apollographql.com/docs/react/caching/cache-configuration/#configuring-the-cache) - `baseUrl` allows us to pass a URL for GraphQL endpoint different from our main endpoint (for example, `${gon.relative_url_root}/api/graphql`) - - `assumeImmutableResults` (set to `false` by default) - this setting, when set to `true`, assumes that every single operation on updating Apollo Cache is immutable. It also sets `freezeResults` to `true`, so any attempt on mutating Apollo Cache throws a console warning in development environment. Please ensure you're following the immutability pattern on cache update operations before setting this option to `true`. - `fetchPolicy` determines how you want your component to interact with the Apollo cache. Defaults to "cache-first". ### Multiple client queries for the same object @@ -179,7 +178,7 @@ with a **new and updated** object. To facilitate the process of updating the cache and returning the new object we use the library [Immer](https://immerjs.github.io/immer/). -When possible, follow these conventions: +Please, follow these conventions: - The updated cache is named `data`. - The original cache data is named `sourceData`. @@ -204,11 +203,6 @@ client.writeQuery({ As shown in the code example by using `produce`, we can perform any kind of direct manipulation of the `draftState`. Besides, `immer` guarantees that a new state which includes the changes to `draftState` is generated. -Finally, to verify whether the immutable cache update is working properly, we need to change -`assumeImmutableResults` to `true` in the default client configuration. See [Apollo Client](#apollo-client) for more information. - -If everything is working properly `assumeImmutableResults` should remain set to `true`. - ## Usage in Vue To use Vue Apollo, import the [Vue Apollo](https://github.com/vuejs/vue-apollo) plugin as well @@ -1106,17 +1100,15 @@ To test the logic of Apollo cache updates, we might want to mock an Apollo Clien To separate tests with mocked client from 'usual' unit tests, create an additional factory and pass the created `mockApollo` as an option to the `createComponent`-factory. This way we only create Apollo Client instance when it's necessary. -We need to inject `VueApollo` to the Vue local instance and, likewise, it is recommended to call `localVue.use()` in `createMockApolloProvider()` to only load it when it is necessary. +We need to inject `VueApollo` into the Vue instance by calling `Vue.use(VueApollo)`. This will install `VueApollo` globally for all the tests in the file. It is recommended to call `Vue.use(VueApollo)` just after the imports. ```javascript import VueApollo from 'vue-apollo'; -import { createLocalVue } from '@vue/test-utils'; +import Vue from 'vue'; -const localVue = createLocalVue(); +Vue.use(VueApollo); function createMockApolloProvider() { - localVue.use(VueApollo); - return createMockApollo(requestHandlers); } @@ -1124,7 +1116,6 @@ function createComponent(options = {}) { const { mockApollo } = options; ... return shallowMount(..., { - localVue, apolloProvider: mockApollo, ... }); @@ -1194,7 +1185,6 @@ describe('Some component', () => { const { mockApollo } = options; return shallowMount(Index, { - localVue, apolloProvider: mockApollo, }); } @@ -1281,7 +1271,6 @@ function createComponent(options = {}) { const { mockApollo } = options; return shallowMount(Index, { - localVue, apolloProvider: mockApollo, }); } @@ -1414,7 +1403,6 @@ const createComponentWithApollo = ({ props = {} } = {}) => { mockApollo = createMockApollo([], mockResolvers); // resolvers are the second parameter wrapper = shallowMount(MyComponent, { - localVue, propsData: {}, apolloProvider: mockApollo, // ... @@ -1482,7 +1470,6 @@ function createComponent(options = {}) { const { mockApollo } = options; return shallowMount(Index, { - localVue, apolloProvider: mockApollo, }); } diff --git a/doc/development/fe_guide/storybook.md b/doc/development/fe_guide/storybook.md index a46157d2cad..9c4bcf02389 100644 --- a/doc/development/fe_guide/storybook.md +++ b/doc/development/fe_guide/storybook.md @@ -47,8 +47,9 @@ To add a story: 1. Write the story as per the [official Storybook instructions](https://storybook.js.org/docs/vue/writing-stories/introduction/) Notes: - - Specify the `title` field of the story as the component's file path from the `javascripts/` directory, - e.g. if the component is located at `app/assets/javascripts/vue_shared/components/sidebar/todo_toggle/todo_button.vue`, specify the story `title` as `vue_shared/components/sidebar/todo_toggle/todo_button`. This will ensure the Storybook navigation maps closely to our internal directory structure. + - Specify the `title` field of the story as the component's file path from the `javascripts/` directory. + + For example, if the component is located at `app/assets/javascripts/vue_shared/components/sidebar/todo_toggle/todo_button.vue`, specify the story `title` as `vue_shared/components/sidebar/todo_toggle/todo_button`. This will ensure the Storybook navigation maps closely to our internal directory structure. ## Mock backend APIs diff --git a/doc/development/fe_guide/style/scss.md b/doc/development/fe_guide/style/scss.md index ffaaa3e87c7..5d5b250e9a9 100644 --- a/doc/development/fe_guide/style/scss.md +++ b/doc/development/fe_guide/style/scss.md @@ -133,6 +133,40 @@ Before adding a new variable for a color or a size, guarantee: - There isn't an existing one. - There isn't a similar one we can use instead. +### Using `extend` at-rule + +Usage of the `extend` at-rule is prohibited due to [memory leaks](https://gitlab.com/gitlab-org/gitlab/-/issues/323021) and [the rule doesn't work as it should to](https://sass-lang.com/documentation/breaking-changes/extend-compound). Use mixins instead: + +```scss +// Bad +.gl-pt-3 { + padding-top: 12px; +} + +.my-element { + @extend .gl-pt-3; +} + +// compiles to +.gl-pt-3, .my-element { + padding-top: 12px; +} + +// Good +@mixing gl-pt-3 { + padding-top: 12px; +} + +.my-element { + @include gl-pt-3; +} + +// compiles to +.my-element { + padding-top: 12px; +} +``` + ## Linting We use [stylelint](https://stylelint.io) to check for style guide conformity. It uses the diff --git a/doc/development/fe_guide/vue.md b/doc/development/fe_guide/vue.md index 509e2f4b688..5d5d37e0398 100644 --- a/doc/development/fe_guide/vue.md +++ b/doc/development/fe_guide/vue.md @@ -73,9 +73,8 @@ component, is that you avoid the need to create a fixture or an HTML element in ##### provide/inject Vue supports dependency injection through [provide/inject](https://vuejs.org/v2/api/#provide-inject). -Values passed to the component through `provide` can be accessed in the component the `inject` configuration. -In the following example of a Vue app initialization, a value from HAML is passed to the component -through the `provide` configuration: +In the component the `inject` configuration accesses the values `provide` passes down. +This example of a Vue app initialization shows how the `provide` configuration passes a value from HAML to the component: ```javascript #js-vue-app{ data: { endpoint: 'foo' }} @@ -251,7 +250,7 @@ return new Vue({ render(createElement) { return createElement('my-component', { props: { - username: gon.current_username, + avatarUrl: gl.avatarUrl, }, }); }, diff --git a/doc/development/fe_guide/vue3_migration.md b/doc/development/fe_guide/vue3_migration.md index 7da462a5f89..c6f480deb22 100644 --- a/doc/development/fe_guide/vue3_migration.md +++ b/doc/development/fe_guide/vue3_migration.md @@ -29,6 +29,12 @@ Component's computed properties / methods or external helpers. `$on`, `$once`, and `$off` methods [are removed](https://github.com/vuejs/rfcs/blob/master/active-rfcs/0020-events-api-change.md) from the Vue instance, so in Vue 3 it can't be used to create an event hub. +**When to use** + +If you are in a Vue app that doesn't use any event hub, try to avoid adding a new one unless absolutely necessary. For example, if you need a child component to react to its parent's event, it's preferred to pass a prop down. Then, use the watch property on that prop in the child component to create the desired side effect. + +If you need cross-component communication (between different Vue apps), then perhaps introducing a hub is the right decision. + **What to use instead** Vue documentation recommends using the [mitt](https://github.com/developit/mitt) library. It's relatively small (200 bytes, compressed) and has a clear API: diff --git a/doc/development/feature_categorization/index.md b/doc/development/feature_categorization/index.md index 20325facc75..d6b64001e13 100644 --- a/doc/development/feature_categorization/index.md +++ b/doc/development/feature_categorization/index.md @@ -96,7 +96,7 @@ second argument: ```ruby class DashboardController < ApplicationController - feature_category :issue_tracking, [:issues, :issues_calendar] + feature_category :team_planning, [:issues, :issues_calendar] feature_category :code_review, [:merge_requests] end ``` @@ -137,7 +137,7 @@ Grape API endpoints can use the `feature_category` class method, like ```ruby module API class Issues < ::API::Base - feature_category :issue_tracking + feature_category :team_planning end end ``` diff --git a/doc/development/feature_flags/controls.md b/doc/development/feature_flags/controls.md index 35dbc2703f9..abb100c659e 100644 --- a/doc/development/feature_flags/controls.md +++ b/doc/development/feature_flags/controls.md @@ -90,9 +90,9 @@ This depends on the feature and what sort of impact it might have. Guidelines: -1. If the feature meets the requirements for creating a [Change Management](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process) issue, create a Change Management issue per [criticality guidelines](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-request-workflows). -1. For simple, low-risk, easily reverted features, proceed and [enable the feature in `#production`](#process). -1. For features that impact the user experience, consider notifying `#support_gitlab-com` beforehand. +- Consider notifying `#support_gitlab-com` beforehand. So in case if the feature has any side effects on user experience, they can mitigate and disable the feature flag to reduce some impact. +- If the feature meets the requirements for creating a [Change Management](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#feature-flags-and-the-change-management-process) issue, create a Change Management issue per [criticality guidelines](https://about.gitlab.com/handbook/engineering/infrastructure/change-management/#change-request-workflows). +- For simple, low-risk, easily reverted features, proceed and [enable the feature in `#production`](#process). #### Process diff --git a/doc/development/feature_flags/index.md b/doc/development/feature_flags/index.md index 987ff7c9fe8..86dc4c73062 100644 --- a/doc/development/feature_flags/index.md +++ b/doc/development/feature_flags/index.md @@ -535,7 +535,7 @@ to ensure the feature works properly. When using the testing environment, all feature flags are enabled by default. WARNING: -This does not apply to end-to-end (QA) tests, which [do not disable feature flags by default](#end-to-end-qa-tests). There is a different [process for using feature flags in end-to-end tests](../testing_guide/end_to_end/feature_flags.md). +This does not apply to end-to-end (QA) tests, which [do not enable feature flags by default](#end-to-end-qa-tests). There is a different [process for using feature flags in end-to-end tests](../testing_guide/end_to_end/feature_flags.md). To disable a feature flag in a test, use the `stub_feature_flags` helper. For example, to globally disable the `ci_live_trace` feature diff --git a/doc/development/gemfile.md b/doc/development/gemfile.md index e071aa4ffd9..5ff1bc7b127 100644 --- a/doc/development/gemfile.md +++ b/doc/development/gemfile.md @@ -55,8 +55,10 @@ to a gem, go through these steps: - For an example, see the [merge request !57805](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/57805). 1. Once the gem is stable - we have been using it in production for a while with few, if any, changes - extract to its own project under - the `gitlab-org` namespace. - 1. When creating the project, follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/#creating-a-new-project). + the [`gitlab-org/ruby/gems` namespace](https://gitlab.com/gitlab-org/ruby/gems/). + + - To create this project: + 1. Follow the [instructions for new projects](https://about.gitlab.com/handbook/engineering/#creating-a-new-project). 1. Follow the instructions for setting up a [CI/CD configuration](https://about.gitlab.com/handbook/engineering/#cicd-configuration). 1. Follow the instructions for [publishing a project](https://about.gitlab.com/handbook/engineering/#publishing-a-project). - See [issue diff --git a/doc/development/gitaly.md b/doc/development/gitaly.md index fa4fa59e35d..5a7a5a6abcb 100644 --- a/doc/development/gitaly.md +++ b/doc/development/gitaly.md @@ -53,7 +53,7 @@ immediately after the Gitaly one. This allows you to test your changes before they are merged. - See [below](#running-tests-with-a-locally-modified-version-of-gitaly) for instructions on running GitLab tests with a modified version of Gitaly. -- In GDK run `gdk install` and restart `gdk run` (or `gdk run app`) to use a locally modified Gitaly version for development +- In GDK run `gdk install` and restart GDK using `gdk restart` to use a locally modified Gitaly version for development ### `gitaly-ruby` diff --git a/doc/development/go_guide/go_upgrade.md b/doc/development/go_guide/go_upgrade.md index 75a093a55ac..53f2d7d176a 100644 --- a/doc/development/go_guide/go_upgrade.md +++ b/doc/development/go_guide/go_upgrade.md @@ -89,10 +89,14 @@ if you need help finding the correct person or labels: 1. Schedule an update with the [GitLab Development Kit](https://gitlab.com/gitlab-org/gitlab-development-kit/-/issues): - Title the issue `Support using Go version <VERSION_NUMBER>`. - Set the issue as related to every issue created in the previous step. -1. Schedule one issue per Secure Stage team and add the `devops::secure` label to each: +1. Schedule one issue per Sec Section team that maintains Go based Security Analyzers and add the `section::sec` label to each: - [Static Analysis tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). - [Composition Analysis tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). - [Container Security tracker](https://gitlab.com/gitlab-org/gitlab/-/issues). + + NOTE: + Updates to these Security analyzers should not block upgrades to Charts or Omnibus since + the analyzers are built independently as separate container images. 1. Schedule builder updates with Distribution projects: - Dependency and GitLab Development Kit issues created in previous steps should be set as blockers. - Each issue should have the title `Support building with Go <VERSION_NUMBER>` and description as noted: diff --git a/doc/development/go_guide/index.md b/doc/development/go_guide/index.md index 44f8924be04..9bf8b7ef89a 100644 --- a/doc/development/go_guide/index.md +++ b/doc/development/go_guide/index.md @@ -364,8 +364,7 @@ JSON format, as all our infrastructure assumes that. When using [Logrus](https://github.com/sirupsen/logrus) you can turn on structured logging simply by using the build in [JSON formatter](https://github.com/sirupsen/logrus#formatters). This follows the -same logging type we use in our [Ruby -applications](../logging.md#use-structured-json-logging). +same logging type we use in our [Ruby applications](../logging.md#use-structured-json-logging). #### How to use Logrus @@ -451,6 +450,78 @@ The conventional Secure [analyzer](https://gitlab.com/gitlab-org/security-produc If the scanner report is small, less than 35 lines, then feel free to [inline the report](https://gitlab.com/gitlab-org/security-products/analyzers/sobelow/-/blob/8bd2428a/convert/convert_test.go#L13-77) rather than use a `testdata` directory. +#### Test Diffs + +The [go-cmp]<https://github.com/google/go-cmp> package should be used when comparing large structs in tests. It makes it possible to output a specific diff where the two structs differ, rather than seeing the whole of both structs printed out in the test logs. Here is a small example: + +```golang +package main + +import ( + "reflect" + "testing" + + "github.com/google/go-cmp/cmp" +) + +type Foo struct { + Desc Bar + Point Baz +} + +type Bar struct { + A string + B string +} + +type Baz struct { + X int + Y int +} + +func TestHelloWorld(t *testing.T) { + want := Foo{ + Desc: Bar{A: "a", B: "b"}, + Point: Baz{X: 1, Y: 2}, + } + + got := Foo{ + Desc: Bar{A: "a", B: "b"}, + Point: Baz{X: 2, Y: 2}, + } + + t.Log("reflect comparison:") + if !reflect.DeepEqual(got, want) { + t.Errorf("Wrong result. want:\n%v\nGot:\n%v", want, got) + } + + t.Log("cmp comparison:") + if diff := cmp.Diff(want, got); diff != "" { + t.Errorf("Wrong result. (-want +got):\n%s", diff) + } +} +``` + +The output demonstrates why `go-cmp` is far superior when comparing large structs. Even though you could spot the difference with this small difference, it quickly gets unwieldy as the data grows. + +```plaintext + main_test.go:36: reflect comparison: + main_test.go:38: Wrong result. want: + {{a b} {1 2}} + Got: + {{a b} {2 2}} + main_test.go:41: cmp comparison: + main_test.go:43: Wrong result. (-want +got): + main.Foo{ + Desc: {A: "a", B: "b"}, + Point: main.Baz{ + - X: 1, + + X: 2, + Y: 2, + }, + } +``` + --- [Return to Development documentation](../index.md). diff --git a/doc/development/i18n/translation.md b/doc/development/i18n/translation.md index 34b7f5e8763..04a9f68abec 100644 --- a/doc/development/i18n/translation.md +++ b/doc/development/i18n/translation.md @@ -48,10 +48,10 @@ Be sure to check the following guidelines before you translate any strings. ### Namespaced strings -When an externalized string is prepended with a namespace (for example, -`s_('OpenedNDaysAgo|Opened')`), the namespace should be removed from the final translation. For -example, in French, `OpenedNDaysAgo|Opened` is translated to `Ouvert•e`, not -`OpenedNDaysAgo|Ouvert•e`. +A namespace precedes the string and is separated from it by a `|` (`namespace|string`). When you see +a namespace before an externalized string, you should remove the namespace from the final +translation. For example, in `OpenedNDaysAgo|Opened`, remove `OpenedNDaysAgo|`. If translating to +French, translate `OpenedNDaysAgo|Opened` to `Ouvert•e`, not `OpenedNDaysAgo|Ouvert•e`. ### Technical terms @@ -61,6 +61,12 @@ should always be in English are noted in the glossary when using This helps maintain a logical connection and consistency between tools (for example, a Git client) and GitLab. +To find the list of technical terms: + +1. Go to [`translate.gitlab.com`](https://translate.gitlab.com). +1. Select the language to translate. +1. Select **Glossary**. + ### Formality The level of formality used in software varies by language: diff --git a/doc/development/index.md b/doc/development/index.md index 3780c986367..fa49d43d46c 100644 --- a/doc/development/index.md +++ b/doc/development/index.md @@ -122,7 +122,7 @@ In these cases, use the following workflow: - [User Experience (UX)](https://about.gitlab.com/handbook/engineering/ux/) - [Security](https://about.gitlab.com/handbook/engineering/security/) - [Quality](https://about.gitlab.com/handbook/engineering/quality/) - - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity-team/) + - [Engineering Productivity](https://about.gitlab.com/handbook/engineering/quality/engineering-productivity/) - [Infrastructure](https://about.gitlab.com/handbook/engineering/infrastructure/) - [Technical Writing](https://about.gitlab.com/handbook/engineering/ux/technical-writing/) @@ -181,6 +181,7 @@ the [reviewer values](https://about.gitlab.com/handbook/engineering/workflow/rev - [Avoid modules with instance variables](module_with_instance_variables.md), if possible - [Guidelines for reusing abstractions](reusing_abstractions.md) +- [Ruby 3 gotchas](ruby3_gotchas.md) ### Rails Framework related diff --git a/doc/development/internal_api.md b/doc/development/internal_api.md index 660d8c60ba8..7790a5e23e6 100644 --- a/doc/development/internal_api.md +++ b/doc/development/internal_api.md @@ -1,829 +1,9 @@ --- -stage: Create -group: Source Code -info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, api +redirect_to: 'internal_api/index.md' +remove_date: '2022-02-09' --- -# Internal API **(FREE)** +This document was moved to [another location](internal_api/index.md). -The internal API is used by different GitLab components, it can not be -used by other consumers. This documentation is intended for people -working on the GitLab codebase. - -This documentation does not yet include the internal API used by -GitLab Pages. - -## Adding new endpoints - -API endpoints should be externally accessible by default, with proper authentication and authorization. -Before adding a new internal endpoint, consider if the API would potentially be -useful to the wider GitLab community and can be made externally accessible. - -One reason we might favor internal API endpoints sometimes is when using such an endpoint requires -internal data that external actors can not have. For example, in the internal Pages API we might use -a secret token that identifies a request as internal or sign a request with a public key that is -not available to a wider community. - -Another reason to separate something into an internal API is when request to such API endpoint -should never go through an edge (public) load balancer. This way we can configure different rate -limiting rules and policies around how the endpoint is being accessed, because we know that only -internal requests can be made to that endpoint going through an internal load balancer. - -## Authentication - -These methods are all authenticated using a shared secret. This secret -is stored in a file at the path configured in `config/gitlab.yml` by -default this is in the root of the rails app named -`.gitlab_shell_secret` - -To authenticate using that token, clients read the contents of that -file, and include the token Base64 encoded in a `secret_token` parameter -or in the `Gitlab-Shared-Secret` header. - -NOTE: -The internal API used by GitLab Pages, and GitLab Kubernetes Agent Server (`kas`) uses JSON Web Token (JWT) -authentication, which is different from GitLab Shell. - -## Git Authentication - -This is called by [Gitaly](https://gitlab.com/gitlab-org/gitaly) and -[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) to check access to a -repository. - -- **When called from GitLab Shell**: No changes are passed, and the internal - API replies with the information needed to pass the request on to Gitaly. -- **When called from Gitaly in a `pre-receive` hook**: The changes are passed - and validated to determine if the push is allowed. - -Calls are limited to 50 seconds each. - -```plaintext -POST /internal/allowed -``` - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `key_id` | string | no | ID of the SSH-key used to connect to GitLab Shell | -| `username` | string | no | Username from the certificate used to connect to GitLab Shell | -| `project` | string | no (if `gl_repository` is passed) | Path to the project | -| `gl_repository` | string | no (if `project` is passed) | Repository identifier, such as `project-7` | -| `protocol` | string | yes | SSH when called from GitLab Shell, HTTP or SSH when called from Gitaly | -| `action` | string | yes | Git command being run (`git-upload-pack`, `git-receive-pack`, `git-upload-archive`) | -| `changes` | string | yes | `<oldrev> <newrev> <refname>` when called from Gitaly, the magic string `_any` when called from GitLab Shell | -| `check_ip` | string | no | IP address from which call to GitLab Shell was made | - -Example request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ - --data "key_id=11&project=gnuwget/wget2&action=git-upload-pack&protocol=ssh" \ - "http://localhost:3001/api/v4/internal/allowed" -``` - -Example response: - -```json -{ - "status": true, - "gl_repository": "project-3", - "gl_project_path": "gnuwget/wget2", - "gl_id": "user-1", - "gl_username": "root", - "git_config_options": [], - "gitaly": { - "repository": { - "storage_name": "default", - "relative_path": "@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git", - "git_object_directory": "", - "git_alternate_object_directories": [], - "gl_repository": "project-3", - "gl_project_path": "gnuwget/wget2" - }, - "address": "unix:/Users/bvl/repos/gitlab/gitaly.socket", - "token": null - }, - "gl_console_messages": [] -} -``` - -### Known consumers - -- Gitaly -- GitLab Shell - -## LFS Authentication - -This is the endpoint that gets called from GitLab Shell to provide -information for LFS clients when the repository is accessed over SSH. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `key_id` | string | no | ID of the SSH-key used to connect to GitLab Shell | -| `username`| string | no | Username from the certificate used to connect to GitLab Shell | -| `project` | string | no | Path to the project | - -Example request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ - --data "key_id=11&project=gnuwget/wget2" "http://localhost:3001/api/v4/internal/lfs_authenticate" -``` - -```json -{ - "username": "root", - "lfs_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkYXRhIjp7ImFjdG9yIjoicm9vdCJ9LCJqdGkiOiIyYWJhZDcxZC0xNDFlLTQ2NGUtOTZlMi1mODllYWRiMGVmZTYiLCJpYXQiOjE1NzAxMTc2NzYsIm5iZiI6MTU3MDExNzY3MSwiZXhwIjoxNTcwMTE5NDc2fQ.g7atlBw1QMY7QEBVPE0LZ8ZlKtaRzaMRmNn41r2YITM", - "repository_http_path": "http://localhost:3001/gnuwget/wget2.git", - "expires_in": 1800 -} -``` - -### Known consumers - -- GitLab Shell - -## Authorized Keys Check - -This endpoint is called by the GitLab Shell authorized keys -check. Which is called by OpenSSH for [fast SSH key -lookup](../administration/operations/fast_ssh_key_lookup.md). - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `key` | string | yes | SSH key as passed by OpenSSH to GitLab Shell | - -```plaintext -GET /internal/authorized_keys -``` - -Example request: - -```shell -curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/authorized_keys?key=<key as passed by OpenSSH>" -``` - -Example response: - -```json -{ - "id": 11, - "title": "admin@example.com", - "key": "ssh-rsa ...", - "created_at": "2019-06-27T15:29:02.219Z" -} -``` - -### Known consumers - -- GitLab Shell - -## Get user for user ID or key - -This endpoint is used when a user performs `ssh git@gitlab.com`. It -discovers the user associated with an SSH key. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | -| `username` | string | no | Username of the user being looked up, used by GitLab Shell when authenticating using a certificate | - -```plaintext -GET /internal/discover -``` - -Example request: - -```shell -curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/discover?key_id=7" -``` - -Example response: - -```json -{ - "id": 7, - "name": "Dede Eichmann", - "username": "rubi" -} -``` - -### Known consumers - -- GitLab Shell - -## Instance information - -This gets some generic information about the instance. This is used -by Geo nodes to get information about each other. - -```plaintext -GET /internal/check -``` - -Example request: - -```shell -curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/check" -``` - -Example response: - -```json -{ - "api_version": "v4", - "gitlab_version": "12.3.0-pre", - "gitlab_rev": "d69c988e6a6", - "redis": true -} -``` - -### Known consumers - -- GitLab Geo -- GitLab Shell's `bin/check` -- Gitaly - -## Get new 2FA recovery codes using an SSH key - -This is called from GitLab Shell and allows users to get new 2FA -recovery codes based on their SSH key. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | -| `user_id` | integer | no | **Deprecated** User_id for which to generate new recovery codes | - -```plaintext -GET /internal/two_factor_recovery_codes -``` - -Example request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ - --data "key_id=7" "http://localhost:3001/api/v4/internal/two_factor_recovery_codes" -``` - -Example response: - -```json -{ - "success": true, - "recovery_codes": [ - "d93ee7037944afd5", - "19d7b84862de93dd", - "1e8c52169195bf71", - "be50444dddb7ca84", - "26048c77d161d5b7", - "482d5c03d1628c47", - "d2c695e309ce7679", - "dfb4748afc4f12a7", - "0e5f53d1399d7979", - "af04d5622153b020" - ] -} -``` - -### Known consumers - -- GitLab Shell - -## Get new personal access-token - -This is called from GitLab Shell and allows users to generate a new -personal access token. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `name` | string | yes | The name of the new token | -| `scopes` | string array | yes | The authorization scopes for the new token, these must be valid token scopes | -| `expires_at` | string | no | The expiry date for the new token | -| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | -| `user_id` | integer | no | User\_id for which to generate the new token | - -```plaintext -POST /internal/personal_access_token -``` - -Example request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ - --data "user_id=29&name=mytokenname&scopes[]=read_user&scopes[]=read_repository&expires_at=2020-07-24" \ - "http://localhost:3001/api/v4/internal/personal_access_token" -``` - -Example response: - -```json -{ - "success": true, - "token": "Hf_79B288hRv_3-TSD1R", - "scopes": ["read_user","read_repository"], - "expires_at": "2020-07-24" -} -``` - -### Known consumers - -- GitLab Shell - -## Incrementing counter on pre-receive - -This is called from the Gitaly hooks increasing the reference counter -for a push that might be accepted. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `gl_repository` | string | yes | repository identifier for the repository receiving the push | - -```plaintext -POST /internal/pre_receive -``` - -Example request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ - --data "gl_repository=project-7" "http://localhost:3001/api/v4/internal/pre_receive" -``` - -Example response: - -```json -{ - "reference_counter_increased": true -} -``` - -## PostReceive - -Called from Gitaly after a receiving a push. This triggers the -`PostReceive`-worker in Sidekiq, processes the passed push options and -builds the response including messages that need to be displayed to -the user. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `identifier` | string | yes | `user-[id]` or `key-[id]` Identifying the user performing the push | -| `gl_repository` | string | yes | identifier of the repository being pushed to | -| `push_options` | string array | no | array of push options | -| `changes` | string | no | refs to be updated in the push in the format `oldrev newrev refname\n`. | - -```plaintext -POST /internal/post_receive -``` - -Example Request: - -```shell -curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ - --data "gl_repository=project-7" --data "identifier=user-1" \ - --data "changes=0000000000000000000000000000000000000000 fd9e76b9136bdd9fe217061b497745792fe5a5ee gh-pages\n" \ - "http://localhost:3001/api/v4/internal/post_receive" -``` - -Example response: - -```json -{ - "messages": [ - { - "message": "Hello from post-receive", - "type": "alert" - } - ], - "reference_counter_decreased": true -} -``` - -## Kubernetes agent endpoints - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41045) in GitLab 13.4. -> - This feature is not deployed on GitLab.com -> - It's not recommended for production use. - -The following endpoints are used by the GitLab Kubernetes Agent Server (`kas`) -for various purposes. - -These endpoints are all authenticated using JWT. The JWT secret is stored in a file -specified in `config/gitlab.yml`. By default, the location is in the root of the -GitLab Rails app in a file called `.gitlab_kas_secret`. - -WARNING: -The Kubernetes agent is under development and is not recommended for production use. - -### Kubernetes agent information - -Called from GitLab Kubernetes Agent Server (`kas`) to retrieve agent -information for the given agent token. This returns the Gitaly connection -information for the agent's project in order for `kas` to fetch and update -the agent's configuration. - -```plaintext -GET /internal/kubernetes/agent_info -``` - -Example Request: - -```shell -curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ - --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info" -``` - -### Kubernetes agent project information - -Called from GitLab Kubernetes Agent Server (`kas`) to retrieve project -information for the given agent token. This returns the Gitaly -connection for the requested project. GitLab `kas` uses this to configure -the agent to fetch Kubernetes resources from the project repository to -sync. - -Only public projects are supported. For private projects, the ability for the -agent to be authorized is [not yet implemented](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../api/index.md#namespaced-path-encoding) | - -```plaintext -GET /internal/kubernetes/project_info -``` - -Example Request: - -```shell -curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ - --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7" -``` - -### Kubernetes agent usage metrics - -Called from GitLab Kubernetes Agent Server (`kas`) to increase the usage -metric counters. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `gitops_sync_count` | integer| no | The number to increase the `gitops_sync_count` counter by | -| `k8s_api_proxy_request_count` | integer| no | The number to increase the `k8s_api_proxy_request_count` counter by | - -```plaintext -POST /internal/kubernetes/usage_metrics -``` - -Example Request: - -```shell -curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Content-Type: application/json" \ - --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" -``` - -### Kubernetes agent alert metrics - -Called from GitLab Kubernetes Agent Server (KAS) to save alerts derived from Cilium on Kubernetes -Cluster. - -| Attribute | Type | Required | Description | -|:----------|:-------|:---------|:------------| -| `alert` | Hash | yes | Alerts detail. Same format as [3rd party alert](../operations/incident_management/integrations.md#customize-the-alert-payload-outside-of-gitlab). | - -```plaintext -POST internal/kubernetes/modules/cilium_alert -``` - -Example Request: - -```shell -curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ - --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ - --data '"{\"alert\":{\"title\":\"minimal\",\"message\":\"network problem\",\"evalMatches\":[{\"value\":1,\"metric\":\"Count\",\"tags\":{}}]}}"' \ - "http://localhost:3000/api/v4/internal/kubernetes/modules/cilium_alert" -``` - -### Create Starboard vulnerability - -Called from the GitLab Kubernetes Agent Server (`kas`) to create a security vulnerability -from a Starboard vulnerability report. This request is idempotent. Multiple requests with the same data -create a single vulnerability. - -| Attribute | Type | Required | Description | -|:----------------|:-------|:---------|:------------| -| `vulnerability` | Hash | yes | Vulnerability data matching the security report schema [`vulnerability` field](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/src/security-report-format.json). | -| `scanner` | Hash | yes | Scanner data matching the security report schema [`scanner` field](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/src/security-report-format.json). | - -```plaintext -PUT internal/kubernetes/modules/starboard_vulnerability -``` - -Example Request: - -```shell -curl --request PUT --header "Gitlab-Kas-Api-Request: <JWT token>" \ - --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ - --url "http://localhost:3000/api/v4/internal/kubernetes/modules/starboard_vulnerability" \ - --data '{ - "vulnerability": { - "name": "CVE-123-4567 in libc", - "severity": "high", - "confidence": "unknown", - "location": { - "kubernetes_resource": { - "namespace": "production", - "kind": "deployment", - "name": "nginx", - "container": "nginx" - } - }, - "identifiers": [ - { - "type": "cve", - "name": "CVE-123-4567", - "value": "CVE-123-4567" - } - ] - }, - "scanner": { - "id": "starboard_trivy", - "name": "Trivy (via Starboard Operator)", - "vendor": "GitLab" - } -}' -``` - -## Subscriptions - -The subscriptions endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) -in order to apply subscriptions including trials, and add-on purchases, for personal namespaces or top-level groups within GitLab.com. - -### Creating a subscription - -Use a POST to create a subscription. - -```plaintext -POST /namespaces/:id/gitlab_subscription -``` - -| Attribute | Type | Required | Description | -|:------------|:--------|:---------|:------------| -| `start_date` | date | yes | Start date of subscription | -| `end_date` | date | no | End date of subscription | -| `plan_code` | string | no | Subscription tier code | -| `seats` | integer | no | Number of seats in subscription | -| `max_seats_used` | integer | no | Highest number of active users in the last month | -| `auto_renew` | boolean | no | Whether subscription auto-renews on end date | -| `trial` | boolean | no | Whether subscription is a trial | -| `trial_starts_on` | date | no | Start date of trial | -| `trial_ends_on` | date | no | End date of trial | - -Example request: - -```shell -curl --request POST --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription?start_date="2020-07-15"&plan="premium"&seats=10" -``` - -Example response: - -```json -{ - "plan": { - "code":"premium", - "name":"premium", - "trial":false, - "auto_renew":null, - "upgradable":false - }, - "usage": { - "seats_in_subscription":10, - "seats_in_use":1, - "max_seats_used":0, - "seats_owed":0 - }, - "billing": { - "subscription_start_date":"2020-07-15", - "subscription_end_date":null, - "trial_ends_on":null - } -} -``` - -### Updating a subscription - -Use a PUT command to update an existing subscription. - -```plaintext -PUT /namespaces/:id/gitlab_subscription -``` - -| Attribute | Type | Required | Description | -|:------------|:--------|:---------|:------------| -| `start_date` | date | no | Start date of subscription | -| `end_date` | date | no | End date of subscription | -| `plan_code` | string | no | Subscription tier code | -| `seats` | integer | no | Number of seats in subscription | -| `max_seats_used` | integer | no | Highest number of active users in the last month | -| `auto_renew` | boolean | no | Whether subscription auto-renews on end date | -| `trial` | boolean | no | Whether subscription is a trial | -| `trial_starts_on` | date | no | Start date of trial. Required if trial is true. | -| `trial_ends_on` | date | no | End date of trial | - -Example request: - -```shell -curl --request PUT --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription?max_seats_used=0" -``` - -Example response: - -```json -{ - "plan": { - "code":"premium", - "name":"premium", - "trial":false, - "auto_renew":null, - "upgradable":false - }, - "usage": { - "seats_in_subscription":80, - "seats_in_use":82, - "max_seats_used":0, - "seats_owed":2 - }, - "billing": { - "subscription_start_date":"2020-07-15", - "subscription_end_date":"2021-07-15", - "trial_ends_on":null - } -} -``` - -### Retrieving a subscription - -Use a GET command to view an existing subscription. - -```plaintext -GET /namespaces/:id/gitlab_subscription -``` - -Example request: - -```shell -curl --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription" -``` - -Example response: - -```json -{ - "plan": { - "code":"premium", - "name":"premium", - "trial":false, - "auto_renew":null, - "upgradable":false - }, - "usage": { - "seats_in_subscription":80, - "seats_in_use":82, - "max_seats_used":82, - "seats_owed":2 - }, - "billing": { - "subscription_start_date":"2020-07-15", - "subscription_end_date":"2021-07-15", - "trial_ends_on":null - } -} -``` - -### Known consumers - -- CustomersDot - -## CI minute provisioning - -The CI Minute endpoints are used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) -to apply additional packs of CI minutes, for personal namespaces or top-level groups within GitLab.com. - -### Creating an additional pack - -Use a POST to create additional packs. - -```plaintext -POST /namespaces/:id/minutes -``` - -| Attribute | Type | Required | Description | -|:------------|:--------|:---------|:------------| -| `packs` | array | yes | An array of purchased minutes packs | -| `packs[expires_at]` | date | yes | Expiry date of the purchased pack| -| `packs[number_of_minutes]` | integer | yes | Number of additional minutes | -| `packs[purchase_xid]` | string | yes | The unique ID of the purchase | - -Example request: - -```shell -curl --request POST \ - --url "http://localhost:3000/api/v4/namespaces/123/minutes" \ - --header 'Content-Type: application/json' \ - --header 'PRIVATE-TOKEN: <admin access token>' \ - --data '{ - "packs": [ - { - "number_of_minutes": 10000, - "expires_at": "2022-01-01", - "purchase_xid": "46952fe69bebc1a4de10b2b4ff439d0c" - } - ] - }' -``` - -Example response: - -```json -[ - { - "namespace_id": 123, - "expires_at": "2022-01-01", - "number_of_minutes": 10000, - "purchase_xid": "46952fe69bebc1a4de10b2b4ff439d0c" - } -] -``` - -### Moving additional packs - -Use a PATCH to move additional packs from one namespace to another. - -```plaintext -PATCH /namespaces/:id/minutes/move/:target_id -``` - -| Attribute | Type | Required | Description | -|:------------|:--------|:---------|:------------| -| `id` | string | yes | The ID of the namespace to transfer packs from | -| `target_id` | string | yes | The ID of the target namespace to transfer the packs to | - -Example request: - -```shell -curl --request PATCH \ - --url "http://localhost:3000/api/v4/namespaces/123/minutes/move/321" \ - --header 'PRIVATE-TOKEN: <admin access token>' -``` - -Example response: - -```json -{ - "message": "202 Accepted" -} -``` - -### Known consumers - -- CustomersDot - -## Upcoming reconciliations - -The `upcoming_reconciliations` endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) -to update upcoming reconciliations for namespaces. - -### Updating `upcoming_reconciliations` - -Use a PUT command to update `upcoming_reconciliations`. - -```plaintext -PUT /internal/upcoming_reconciliations -``` - -| Attribute | Type | Required | Description | -|:-------------------|:-----------|:---------|:------------| -| `upcoming_reconciliations` | array | yes | Array of upcoming reconciliations | - -Each array element contains: - -| Attribute | Type | Required | Description | -|:-------------------|:-----------|:---------|:------------| -| `namespace_id` | integer | yes | ID of the namespace to be reconciled | -| `next_reconciliation_date` | date | yes | Date when next reconciliation will happen | -| `display_alert_from` | date | yes | Start date to display alert of upcoming reconciliation | - -Example request: - -```shell -curl --request PUT --header "PRIVATE-TOKEN: <admin_access_token>" --header "Content-Type: application/json" \ - --data '{"upcoming_reconciliations": [{"namespace_id": 127, "next_reconciliation_date": "13 Jun 2021", "display_alert_from": "06 Jun 2021"}, {"namespace_id": 129, "next_reconciliation_date": "12 Jun 2021", "display_alert_from": "05 Jun 2021"}]}' \ - "https://gitlab.com/api/v4/internal/upcoming_reconciliations" -``` - -Example response: - -```plaintext -200 -``` - -### Known consumers - -- CustomersDot +<!-- This redirect file can be deleted after <2022-02-09>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/development/internal_api/index.md b/doc/development/internal_api/index.md new file mode 100644 index 00000000000..0fe9a5362cf --- /dev/null +++ b/doc/development/internal_api/index.md @@ -0,0 +1,831 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +type: reference, api +--- + +# Internal API **(FREE)** + +The internal API is used by different GitLab components, it can not be +used by other consumers. This documentation is intended for people +working on the GitLab codebase. + +This documentation does not yet include the internal API used by +GitLab Pages. + +## Adding new endpoints + +API endpoints should be externally accessible by default, with proper authentication and authorization. +Before adding a new internal endpoint, consider if the API would potentially be +useful to the wider GitLab community and can be made externally accessible. + +One reason we might favor internal API endpoints sometimes is when using such an endpoint requires +internal data that external actors can not have. For example, in the internal Pages API we might use +a secret token that identifies a request as internal or sign a request with a public key that is +not available to a wider community. + +Another reason to separate something into an internal API is when request to such API endpoint +should never go through an edge (public) load balancer. This way we can configure different rate +limiting rules and policies around how the endpoint is being accessed, because we know that only +internal requests can be made to that endpoint going through an internal load balancer. + +## Authentication + +These methods are all authenticated using a shared secret. This secret +is stored in a file at the path configured in `config/gitlab.yml` by +default this is in the root of the rails app named +`.gitlab_shell_secret` + +To authenticate using that token, clients read the contents of that +file, and include the token Base64 encoded in a `secret_token` parameter +or in the `Gitlab-Shared-Secret` header. + +NOTE: +The internal API used by GitLab Pages, and GitLab Kubernetes Agent Server (`kas`) uses JSON Web Token (JWT) +authentication, which is different from GitLab Shell. + +## Git Authentication + +This is called by [Gitaly](https://gitlab.com/gitlab-org/gitaly) and +[GitLab Shell](https://gitlab.com/gitlab-org/gitlab-shell) to check access to a +repository. + +- **When called from GitLab Shell**: No changes are passed, and the internal + API replies with the information needed to pass the request on to Gitaly. +- **When called from Gitaly in a `pre-receive` hook**: The changes are passed + and validated to determine if the push is allowed. + +Calls are limited to 50 seconds each. + +This endpoint is covered in more detail on [its own page](internal_api_allowed.md), due to the scope of what it covers. + +```plaintext +POST /internal/allowed +``` + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `key_id` | string | no | ID of the SSH-key used to connect to GitLab Shell | +| `username` | string | no | Username from the certificate used to connect to GitLab Shell | +| `project` | string | no (if `gl_repository` is passed) | Path to the project | +| `gl_repository` | string | no (if `project` is passed) | Repository identifier, such as `project-7` | +| `protocol` | string | yes | SSH when called from GitLab Shell, HTTP or SSH when called from Gitaly | +| `action` | string | yes | Git command being run (`git-upload-pack`, `git-receive-pack`, `git-upload-archive`) | +| `changes` | string | yes | `<oldrev> <newrev> <refname>` when called from Gitaly, the magic string `_any` when called from GitLab Shell | +| `check_ip` | string | no | IP address from which call to GitLab Shell was made | + +Example request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ + --data "key_id=11&project=gnuwget/wget2&action=git-upload-pack&protocol=ssh" \ + "http://localhost:3001/api/v4/internal/allowed" +``` + +Example response: + +```json +{ + "status": true, + "gl_repository": "project-3", + "gl_project_path": "gnuwget/wget2", + "gl_id": "user-1", + "gl_username": "root", + "git_config_options": [], + "gitaly": { + "repository": { + "storage_name": "default", + "relative_path": "@hashed/4e/07/4e07408562bedb8b60ce05c1decfe3ad16b72230967de01f640b7e4729b49fce.git", + "git_object_directory": "", + "git_alternate_object_directories": [], + "gl_repository": "project-3", + "gl_project_path": "gnuwget/wget2" + }, + "address": "unix:/Users/bvl/repos/gitlab/gitaly.socket", + "token": null + }, + "gl_console_messages": [] +} +``` + +### Known consumers + +- Gitaly +- GitLab Shell + +## LFS Authentication + +This is the endpoint that gets called from GitLab Shell to provide +information for LFS clients when the repository is accessed over SSH. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `key_id` | string | no | ID of the SSH-key used to connect to GitLab Shell | +| `username`| string | no | Username from the certificate used to connect to GitLab Shell | +| `project` | string | no | Path to the project | + +Example request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded token>" \ + --data "key_id=11&project=gnuwget/wget2" "http://localhost:3001/api/v4/internal/lfs_authenticate" +``` + +```json +{ + "username": "root", + "lfs_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJkYXRhIjp7ImFjdG9yIjoicm9vdCJ9LCJqdGkiOiIyYWJhZDcxZC0xNDFlLTQ2NGUtOTZlMi1mODllYWRiMGVmZTYiLCJpYXQiOjE1NzAxMTc2NzYsIm5iZiI6MTU3MDExNzY3MSwiZXhwIjoxNTcwMTE5NDc2fQ.g7atlBw1QMY7QEBVPE0LZ8ZlKtaRzaMRmNn41r2YITM", + "repository_http_path": "http://localhost:3001/gnuwget/wget2.git", + "expires_in": 1800 +} +``` + +### Known consumers + +- GitLab Shell + +## Authorized Keys Check + +This endpoint is called by the GitLab Shell authorized keys +check. Which is called by OpenSSH for [fast SSH key +lookup](../../administration/operations/fast_ssh_key_lookup.md). + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `key` | string | yes | SSH key as passed by OpenSSH to GitLab Shell | + +```plaintext +GET /internal/authorized_keys +``` + +Example request: + +```shell +curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/authorized_keys?key=<key as passed by OpenSSH>" +``` + +Example response: + +```json +{ + "id": 11, + "title": "admin@example.com", + "key": "ssh-rsa ...", + "created_at": "2019-06-27T15:29:02.219Z" +} +``` + +### Known consumers + +- GitLab Shell + +## Get user for user ID or key + +This endpoint is used when a user performs `ssh git@gitlab.com`. It +discovers the user associated with an SSH key. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | +| `username` | string | no | Username of the user being looked up, used by GitLab Shell when authenticating using a certificate | + +```plaintext +GET /internal/discover +``` + +Example request: + +```shell +curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/discover?key_id=7" +``` + +Example response: + +```json +{ + "id": 7, + "name": "Dede Eichmann", + "username": "rubi" +} +``` + +### Known consumers + +- GitLab Shell + +## Instance information + +This gets some generic information about the instance. This is used +by Geo nodes to get information about each other. + +```plaintext +GET /internal/check +``` + +Example request: + +```shell +curl --request GET --header "Gitlab-Shared-Secret: <Base64 encoded secret>" "http://localhost:3001/api/v4/internal/check" +``` + +Example response: + +```json +{ + "api_version": "v4", + "gitlab_version": "12.3.0-pre", + "gitlab_rev": "d69c988e6a6", + "redis": true +} +``` + +### Known consumers + +- GitLab Geo +- GitLab Shell's `bin/check` +- Gitaly + +## Get new 2FA recovery codes using an SSH key + +This is called from GitLab Shell and allows users to get new 2FA +recovery codes based on their SSH key. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | +| `user_id` | integer | no | **Deprecated** User_id for which to generate new recovery codes | + +```plaintext +GET /internal/two_factor_recovery_codes +``` + +Example request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "key_id=7" "http://localhost:3001/api/v4/internal/two_factor_recovery_codes" +``` + +Example response: + +```json +{ + "success": true, + "recovery_codes": [ + "d93ee7037944afd5", + "19d7b84862de93dd", + "1e8c52169195bf71", + "be50444dddb7ca84", + "26048c77d161d5b7", + "482d5c03d1628c47", + "d2c695e309ce7679", + "dfb4748afc4f12a7", + "0e5f53d1399d7979", + "af04d5622153b020" + ] +} +``` + +### Known consumers + +- GitLab Shell + +## Get new personal access-token + +This is called from GitLab Shell and allows users to generate a new +personal access token. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `name` | string | yes | The name of the new token | +| `scopes` | string array | yes | The authorization scopes for the new token, these must be valid token scopes | +| `expires_at` | string | no | The expiry date for the new token | +| `key_id` | integer | no | The ID of the SSH key used as found in the authorized-keys file or through the `/authorized_keys` check | +| `user_id` | integer | no | User ID for which to generate the new token | + +```plaintext +POST /internal/personal_access_token +``` + +Example request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "user_id=29&name=mytokenname&scopes[]=read_user&scopes[]=read_repository&expires_at=2020-07-24" \ + "http://localhost:3001/api/v4/internal/personal_access_token" +``` + +Example response: + +```json +{ + "success": true, + "token": "Hf_79B288hRv_3-TSD1R", + "scopes": ["read_user","read_repository"], + "expires_at": "2020-07-24" +} +``` + +### Known consumers + +- GitLab Shell + +## Incrementing counter on pre-receive + +This is called from the Gitaly hooks increasing the reference counter +for a push that might be accepted. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `gl_repository` | string | yes | repository identifier for the repository receiving the push | + +```plaintext +POST /internal/pre_receive +``` + +Example request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "gl_repository=project-7" "http://localhost:3001/api/v4/internal/pre_receive" +``` + +Example response: + +```json +{ + "reference_counter_increased": true +} +``` + +## PostReceive + +Called from Gitaly after a receiving a push. This triggers the +`PostReceive`-worker in Sidekiq, processes the passed push options and +builds the response including messages that need to be displayed to +the user. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `identifier` | string | yes | `user-[id]` or `key-[id]` Identifying the user performing the push | +| `gl_repository` | string | yes | identifier of the repository being pushed to | +| `push_options` | string array | no | array of push options | +| `changes` | string | no | refs to be updated in the push in the format `oldrev newrev refname\n`. | + +```plaintext +POST /internal/post_receive +``` + +Example Request: + +```shell +curl --request POST --header "Gitlab-Shared-Secret: <Base64 encoded secret>" \ + --data "gl_repository=project-7" --data "identifier=user-1" \ + --data "changes=0000000000000000000000000000000000000000 fd9e76b9136bdd9fe217061b497745792fe5a5ee gh-pages\n" \ + "http://localhost:3001/api/v4/internal/post_receive" +``` + +Example response: + +```json +{ + "messages": [ + { + "message": "Hello from post-receive", + "type": "alert" + } + ], + "reference_counter_decreased": true +} +``` + +## Kubernetes agent endpoints + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41045) in GitLab 13.4. +> - This feature is not deployed on GitLab.com +> - It's not recommended for production use. + +The following endpoints are used by the GitLab Kubernetes Agent Server (`kas`) +for various purposes. + +These endpoints are all authenticated using JWT. The JWT secret is stored in a file +specified in `config/gitlab.yml`. By default, the location is in the root of the +GitLab Rails app in a file called `.gitlab_kas_secret`. + +WARNING: +The Kubernetes agent is under development and is not recommended for production use. + +### Kubernetes agent information + +Called from GitLab Kubernetes Agent Server (`kas`) to retrieve agent +information for the given agent token. This returns the Gitaly connection +information for the agent's project in order for `kas` to fetch and update +the agent's configuration. + +```plaintext +GET /internal/kubernetes/agent_info +``` + +Example Request: + +```shell +curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/agent_info" +``` + +### Kubernetes agent project information + +Called from GitLab Kubernetes Agent Server (`kas`) to retrieve project +information for the given agent token. This returns the Gitaly +connection for the requested project. GitLab `kas` uses this to configure +the agent to fetch Kubernetes resources from the project repository to +sync. + +Only public projects are supported. For private projects, the ability for the +agent to be authorized is [not yet implemented](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `id` | integer/string | yes | The ID or [URL-encoded path of the project](../../api/index.md#namespaced-path-encoding) | + +```plaintext +GET /internal/kubernetes/project_info +``` + +Example Request: + +```shell +curl --request GET --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" "http://localhost:3000/api/v4/internal/kubernetes/project_info?id=7" +``` + +### Kubernetes agent usage metrics + +Called from GitLab Kubernetes Agent Server (`kas`) to increase the usage +metric counters. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `gitops_sync_count` | integer| no | The number to increase the `gitops_sync_count` counter by | +| `k8s_api_proxy_request_count` | integer| no | The number to increase the `k8s_api_proxy_request_count` counter by | + +```plaintext +POST /internal/kubernetes/usage_metrics +``` + +Example Request: + +```shell +curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" --header "Content-Type: application/json" \ + --data '{"gitops_sync_count":1}' "http://localhost:3000/api/v4/internal/kubernetes/usage_metrics" +``` + +### Kubernetes agent alert metrics + +Called from GitLab Kubernetes Agent Server (KAS) to save alerts derived from Cilium on Kubernetes +Cluster. + +| Attribute | Type | Required | Description | +|:----------|:-------|:---------|:------------| +| `alert` | Hash | yes | Alerts detail. Same format as [3rd party alert](../../operations/incident_management/integrations.md#customize-the-alert-payload-outside-of-gitlab). | + +```plaintext +POST internal/kubernetes/modules/cilium_alert +``` + +Example Request: + +```shell +curl --request POST --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ + --data '"{\"alert\":{\"title\":\"minimal\",\"message\":\"network problem\",\"evalMatches\":[{\"value\":1,\"metric\":\"Count\",\"tags\":{}}]}}"' \ + "http://localhost:3000/api/v4/internal/kubernetes/modules/cilium_alert" +``` + +### Create Starboard vulnerability + +Called from the GitLab Kubernetes Agent Server (`kas`) to create a security vulnerability +from a Starboard vulnerability report. This request is idempotent. Multiple requests with the same data +create a single vulnerability. + +| Attribute | Type | Required | Description | +|:----------------|:-------|:---------|:------------| +| `vulnerability` | Hash | yes | Vulnerability data matching the security report schema [`vulnerability` field](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/src/security-report-format.json). | +| `scanner` | Hash | yes | Scanner data matching the security report schema [`scanner` field](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/src/security-report-format.json). | + +```plaintext +PUT internal/kubernetes/modules/starboard_vulnerability +``` + +Example Request: + +```shell +curl --request PUT --header "Gitlab-Kas-Api-Request: <JWT token>" \ + --header "Authorization: Bearer <agent token>" --header "Content-Type: application/json" \ + --url "http://localhost:3000/api/v4/internal/kubernetes/modules/starboard_vulnerability" \ + --data '{ + "vulnerability": { + "name": "CVE-123-4567 in libc", + "severity": "high", + "confidence": "unknown", + "location": { + "kubernetes_resource": { + "namespace": "production", + "kind": "deployment", + "name": "nginx", + "container": "nginx" + } + }, + "identifiers": [ + { + "type": "cve", + "name": "CVE-123-4567", + "value": "CVE-123-4567" + } + ] + }, + "scanner": { + "id": "starboard_trivy", + "name": "Trivy (via Starboard Operator)", + "vendor": "GitLab" + } +}' +``` + +## Subscriptions + +The subscriptions endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) +in order to apply subscriptions including trials, and add-on purchases, for personal namespaces or top-level groups within GitLab.com. + +### Creating a subscription + +Use a POST to create a subscription. + +```plaintext +POST /namespaces/:id/gitlab_subscription +``` + +| Attribute | Type | Required | Description | +|:------------|:--------|:---------|:------------| +| `start_date` | date | yes | Start date of subscription | +| `end_date` | date | no | End date of subscription | +| `plan_code` | string | no | Subscription tier code | +| `seats` | integer | no | Number of seats in subscription | +| `max_seats_used` | integer | no | Highest number of active users in the last month | +| `auto_renew` | boolean | no | Whether subscription auto-renews on end date | +| `trial` | boolean | no | Whether subscription is a trial | +| `trial_starts_on` | date | no | Start date of trial | +| `trial_ends_on` | date | no | End date of trial | + +Example request: + +```shell +curl --request POST --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription?start_date="2020-07-15"&plan="premium"&seats=10" +``` + +Example response: + +```json +{ + "plan": { + "code":"premium", + "name":"premium", + "trial":false, + "auto_renew":null, + "upgradable":false + }, + "usage": { + "seats_in_subscription":10, + "seats_in_use":1, + "max_seats_used":0, + "seats_owed":0 + }, + "billing": { + "subscription_start_date":"2020-07-15", + "subscription_end_date":null, + "trial_ends_on":null + } +} +``` + +### Updating a subscription + +Use a PUT command to update an existing subscription. + +```plaintext +PUT /namespaces/:id/gitlab_subscription +``` + +| Attribute | Type | Required | Description | +|:------------|:--------|:---------|:------------| +| `start_date` | date | no | Start date of subscription | +| `end_date` | date | no | End date of subscription | +| `plan_code` | string | no | Subscription tier code | +| `seats` | integer | no | Number of seats in subscription | +| `max_seats_used` | integer | no | Highest number of active users in the last month | +| `auto_renew` | boolean | no | Whether subscription auto-renews on end date | +| `trial` | boolean | no | Whether subscription is a trial | +| `trial_starts_on` | date | no | Start date of trial. Required if trial is true. | +| `trial_ends_on` | date | no | End date of trial | + +Example request: + +```shell +curl --request PUT --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription?max_seats_used=0" +``` + +Example response: + +```json +{ + "plan": { + "code":"premium", + "name":"premium", + "trial":false, + "auto_renew":null, + "upgradable":false + }, + "usage": { + "seats_in_subscription":80, + "seats_in_use":82, + "max_seats_used":0, + "seats_owed":2 + }, + "billing": { + "subscription_start_date":"2020-07-15", + "subscription_end_date":"2021-07-15", + "trial_ends_on":null + } +} +``` + +### Retrieving a subscription + +Use a GET command to view an existing subscription. + +```plaintext +GET /namespaces/:id/gitlab_subscription +``` + +Example request: + +```shell +curl --header "TOKEN: <admin_access_token>" "https://gitlab.com/api/v4/namespaces/1234/gitlab_subscription" +``` + +Example response: + +```json +{ + "plan": { + "code":"premium", + "name":"premium", + "trial":false, + "auto_renew":null, + "upgradable":false + }, + "usage": { + "seats_in_subscription":80, + "seats_in_use":82, + "max_seats_used":82, + "seats_owed":2 + }, + "billing": { + "subscription_start_date":"2020-07-15", + "subscription_end_date":"2021-07-15", + "trial_ends_on":null + } +} +``` + +### Known consumers + +- CustomersDot + +## CI minute provisioning + +The CI Minute endpoints are used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) +to apply additional packs of CI minutes, for personal namespaces or top-level groups within GitLab.com. + +### Creating an additional pack + +Use a POST to create additional packs. + +```plaintext +POST /namespaces/:id/minutes +``` + +| Attribute | Type | Required | Description | +|:------------|:--------|:---------|:------------| +| `packs` | array | yes | An array of purchased minutes packs | +| `packs[expires_at]` | date | yes | Expiry date of the purchased pack| +| `packs[number_of_minutes]` | integer | yes | Number of additional minutes | +| `packs[purchase_xid]` | string | yes | The unique ID of the purchase | + +Example request: + +```shell +curl --request POST \ + --url "http://localhost:3000/api/v4/namespaces/123/minutes" \ + --header 'Content-Type: application/json' \ + --header 'PRIVATE-TOKEN: <admin access token>' \ + --data '{ + "packs": [ + { + "number_of_minutes": 10000, + "expires_at": "2022-01-01", + "purchase_xid": "46952fe69bebc1a4de10b2b4ff439d0c" + } + ] + }' +``` + +Example response: + +```json +[ + { + "namespace_id": 123, + "expires_at": "2022-01-01", + "number_of_minutes": 10000, + "purchase_xid": "46952fe69bebc1a4de10b2b4ff439d0c" + } +] +``` + +### Moving additional packs + +Use a PATCH to move additional packs from one namespace to another. + +```plaintext +PATCH /namespaces/:id/minutes/move/:target_id +``` + +| Attribute | Type | Required | Description | +|:------------|:--------|:---------|:------------| +| `id` | string | yes | The ID of the namespace to transfer packs from | +| `target_id` | string | yes | The ID of the target namespace to transfer the packs to | + +Example request: + +```shell +curl --request PATCH \ + --url "http://localhost:3000/api/v4/namespaces/123/minutes/move/321" \ + --header 'PRIVATE-TOKEN: <admin access token>' +``` + +Example response: + +```json +{ + "message": "202 Accepted" +} +``` + +### Known consumers + +- CustomersDot + +## Upcoming reconciliations + +The `upcoming_reconciliations` endpoint is used by [CustomersDot](https://gitlab.com/gitlab-org/customers-gitlab-com) (`customers.gitlab.com`) +to update upcoming reconciliations for namespaces. + +### Updating `upcoming_reconciliations` + +Use a PUT command to update `upcoming_reconciliations`. + +```plaintext +PUT /internal/upcoming_reconciliations +``` + +| Attribute | Type | Required | Description | +|:-------------------|:-----------|:---------|:------------| +| `upcoming_reconciliations` | array | yes | Array of upcoming reconciliations | + +Each array element contains: + +| Attribute | Type | Required | Description | +|:-------------------|:-----------|:---------|:------------| +| `namespace_id` | integer | yes | ID of the namespace to be reconciled | +| `next_reconciliation_date` | date | yes | Date when next reconciliation will happen | +| `display_alert_from` | date | yes | Start date to display alert of upcoming reconciliation | + +Example request: + +```shell +curl --request PUT --header "PRIVATE-TOKEN: <admin_access_token>" --header "Content-Type: application/json" \ + --data '{"upcoming_reconciliations": [{"namespace_id": 127, "next_reconciliation_date": "13 Jun 2021", "display_alert_from": "06 Jun 2021"}, {"namespace_id": 129, "next_reconciliation_date": "12 Jun 2021", "display_alert_from": "05 Jun 2021"}]}' \ + "https://gitlab.com/api/v4/internal/upcoming_reconciliations" +``` + +Example response: + +```plaintext +200 +``` + +### Known consumers + +- CustomersDot diff --git a/doc/development/internal_api/internal_api_allowed.md b/doc/development/internal_api/internal_api_allowed.md new file mode 100644 index 00000000000..83dbb54babd --- /dev/null +++ b/doc/development/internal_api/internal_api_allowed.md @@ -0,0 +1,109 @@ +--- +stage: Create +group: Source Code +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +type: reference, api +--- + +# Internal allowed API + +The `internal/allowed` endpoint assesses whether a user has permission to perform +certain operations on the Git repository. It performs multiple checks, such as: + +- Ensuring the branch or tag name is acceptable. +- Whether or not the user has the authority to perform that action. + +## Endpoint definition + +The internal API endpoints are defined under +[`lib/api/internal`](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/api/internal), +and the `/allowed` path is in +[`lib/api/internal/base.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/api/internal/base.rb). + +## Use the endpoint + +`internal/allowed` is called when you: + +- Push to the repository. +- Perform actions on the repository through the GitLab user interface, such as + applying suggestions or using the GitLab IDE. + +Gitaly typically calls this endpoint. It is only called internally (by other parts +of the application) rather than by external users of the API. + +## Push checks + +A key part of the `internal/allowed` flow is the call to +`EE::Gitlab::Checks::PushRuleCheck`, which can perform the following checks: + +- `EE::Gitlab::Checks::PushRules::CommitCheck` +- `EE::Gitlab::Checks::PushRules::TagCheck` +- `EE::Gitlab::Checks::PushRules::BranchCheck` +- `EE::Gitlab::Checks::PushRules::FileSizeCheck` + +## Recursion + +Some of the Gitaly RPCs called by `internal/allowed` then, themselves, make calls +back to `internal/allowed`. These calls are now correlated with the original request. +Gitaly relies on the Rails application for authorization, and maintains no permissions model itself. + +These calls show up in the logs differently to the initial requests. {example} + +Because this endpoint can be called recursively, slow performance on this endpoint can result in an exponential performance impact. This documentation is in fact adapted from [the investigation into its performance](https://gitlab.com/gitlab-org/gitlab/-/issues/222247). + +## Known performance issues + +### Refs + +The number of [`refs`](https://git-scm.com/book/en/v2/Git-Internals-Git-References) +on the Git repository have a notable effect on the performance of `git` commands +called upon it. Gitaly RPCs are similarly affected. Certain `git` commands scan +through all refs, causing a notable impact on the speed of those commands. + +On the `internal/allowed` endpoint, the recursive nature of RPC calls mean the +ref counts have an exponential effect on performance. + +#### Environment refs + +[Stale environment refs](https://gitlab.com/gitlab-org/gitlab/-/issues/296625) +are a common example of excessive refs causing performance issues. Stale environment +refs can number into the tens of thousands on busy repositories, as they aren't +cleared up automatically. + +#### Dangling refs + +Dangling refs are created to prevent accidental deletion of objects from object pools. +Large numbers of these refs can exist, which may have potential performance implications. +For existing discussion around this issue, read +[`gitaly#1900`](https://gitlab.com/gitlab-org/gitaly/-/issues/1900). This issue +appears to have less effect than stale environment refs. + +### Pool repositories + +When a fork is created on GitLab, a central pool repository is created and the forks +are linked to it. This pool repository prevents duplication of data by storing +data common to other forks. However, the pool repository is not cleaned up in the +same manner as the standard repositories, and is more prone to the refs issue. + +## Feature flags + +### Parallel push checks + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `parallel_push_checks`. +On GitLab.com, by default this feature is not available. To make it available +per project, ask GitLab.com administrator to +[enable the feature flag](../../administration/feature_flags.md) named `parallel_push_checks`. +You should not use this feature for production environments. + +This experimental feature flag enables the endpoint to run multiple RPCs simultaneously, +reducing the overall time taken by roughly half. This time savings is achieved through +threading, and has potential side effects at large scale. On GitLab.com, this feature flag +is enabled only for `gitlab-org/gitlab` and `gitlab-com/www-gitlab-com` projects. +Without it, those projects routinely time out requests to the endpoint. When this +feature was deployed to all of GitLab.com, some pushes failed, presumably due to +exhausting resources like database connection pools. + +We recommend you enable this feature flag only if you are experiencing timeouts, and +only enable it for that specific project. diff --git a/doc/development/iterating_tables_in_batches.md b/doc/development/iterating_tables_in_batches.md index fba6f2f616d..9ea7fd3dc29 100644 --- a/doc/development/iterating_tables_in_batches.md +++ b/doc/development/iterating_tables_in_batches.md @@ -142,7 +142,7 @@ database query: SELECT "users"."id" FROM "users" ORDER BY "users"."id" ASC LIMIT 1 ``` - + Notice that the query only reads data from the index (`INDEX ONLY SCAN`), the table is not accessed. Database indexes are sorted so taking out the first item is a very cheap operation. @@ -155,7 +155,7 @@ to get a "shifted" `id` value. SELECT "users"."id" FROM "users" WHERE "users"."id" >= 1 ORDER BY "users"."id" ASC LIMIT 1 OFFSET 5 ``` - + Again, the query only looks into the index. The `OFFSET 5` takes out the sixth `id` value: this query reads a maximum of six items from the index regardless of the table size or the iteration @@ -181,7 +181,7 @@ previous iteration in order to find out the next end `id` value. SELECT "users"."id" FROM "users" WHERE "users"."id" >= 302 ORDER BY "users"."id" ASC LIMIT 1 OFFSET 5 ``` - + Now we can easily construct the `users` query for the second iteration. diff --git a/doc/development/jh_features_review.md b/doc/development/jh_features_review.md index cb0f8ddbd13..858048c1e7c 100644 --- a/doc/development/jh_features_review.md +++ b/doc/development/jh_features_review.md @@ -23,6 +23,14 @@ We have two kinds of changes related to JH: If needed, review the corresponding JH merge request located at [JH repository](https://gitlab.com/gitlab-jh/gitlab) +## When to merge files to the GitLab Inc. repository + +Files that are added to the `gitlab-jh` repository outside of `jh/` must be mirrored in the GitLab Inc. repository. + +If code that is added to the GitLab Inc. repository references (for example, `render_if_exists`) any `gitlab-jh` file that does not +exist in the GitLab Inc. codebase, add a comment with a link to the JiHu merge request or file. This is to prevent +the reference from being misidentified as a missing partial and subsequently deleted in the `gitlab` codebase. + ## Process overview See the [merge request process](https://about.gitlab.com/handbook/ceo/chief-of-staff-team/jihu-support/#merge-request-process) diff --git a/doc/development/maintenance_mode.md b/doc/development/maintenance_mode.md index e308ab26c27..c2fd4bab605 100644 --- a/doc/development/maintenance_mode.md +++ b/doc/development/maintenance_mode.md @@ -13,7 +13,7 @@ GitLab Maintenance Mode **only** blocks writes from HTTP and SSH requests at the - [the read-only database method](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/lib/ee/gitlab/database.rb#L13), which toggles special behavior when we are not allowed to write to the database. [Search the codebase for `Gitlab::Database.read_only?`.](https://gitlab.com/search?search=Gitlab%3A%3ADatabase.read_only%3F&group_id=9970&project_id=278964&scope=blobs&search_code=false&snippets=false&repository_ref=) - [the read-only middleware](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/ee/gitlab/middleware/read_only/controller.rb), where HTTP requests that cause database writes are blocked, unless explicitly allowed. -- [Git push access via SSH is denied](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/lib/ee/gitlab/git_access.rb#L13) by returning 401 when `gitlab-shell` POSTs to [`/internal/allowed`](internal_api.md) to [check if access is allowed](internal_api.md#git-authentication). +- [Git push access via SSH is denied](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/lib/ee/gitlab/git_access.rb#L13) by returning 401 when `gitlab-shell` POSTs to [`/internal/allowed`](internal_api/index.md) to [check if access is allowed](internal_api/index.md#git-authentication). - [Container registry authentication service](https://gitlab.com/gitlab-org/gitlab/-/blob/2425e9de50c678413ceaad6ee3bf66f42b7e228c/ee/app/services/ee/auth/container_registry_authentication_service.rb#L12), where updates to the container registry are blocked. The database itself is not in read-only mode (except in a Geo secondary site) and can be written by sources other than the ones blocked. diff --git a/doc/development/migration_style_guide.md b/doc/development/migration_style_guide.md index e03b96a0e14..0bd9979e790 100644 --- a/doc/development/migration_style_guide.md +++ b/doc/development/migration_style_guide.md @@ -135,6 +135,7 @@ various database operations, such as: - [dropping and renaming columns](avoiding_downtime_in_migrations.md#dropping-columns) - [changing column constraints and types](avoiding_downtime_in_migrations.md#changing-column-constraints) - [adding and dropping indexes, tables, and foreign keys](avoiding_downtime_in_migrations.md#adding-indexes) +- [migrating `integer` primary keys to `bigint`](avoiding_downtime_in_migrations.md#migrating-integer-primary-keys-to-bigint) and explains how to perform them without requiring downtime. @@ -312,12 +313,8 @@ A better strategy is to split the migration, so that we only need to acquire one ```ruby enable_lock_retries! -def up - remove_column :users, :full_name -end - -def down - add_column :users, :full_name, :string +def change + remove_column :users, :full_name, :string end ``` @@ -587,8 +584,6 @@ class like so: ```ruby class MyMigration < Gitlab::Database::Migration[1.0] - include Gitlab::Database::MigrationHelpers - disable_ddl_transaction! INDEX_NAME = 'index_name' @@ -632,8 +627,6 @@ be used with a name option. For example: ```ruby class MyMigration < Gitlab::Database::Migration[1.0] - include Gitlab::Database::MigrationHelpers - INDEX_NAME = 'index_name' def up diff --git a/doc/development/multi_version_compatibility.md b/doc/development/multi_version_compatibility.md index f834f4f4ee3..27a7cd6e85e 100644 --- a/doc/development/multi_version_compatibility.md +++ b/doc/development/multi_version_compatibility.md @@ -49,7 +49,15 @@ Is it ok that some nodes have the new Rails version, but some nodes have the old ## A walkthrough of an update -Backwards compatibility problems during updates are often very subtle. This is why it is worth familiarizing yourself with [update instructions](../update/index.md), [reference architectures](../administration/reference_architectures/index.md), and [GitLab.com's architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/). But to illustrate how these problems arise, take a look at this example of a simple update. +Backward compatibility problems during updates are often very subtle. This is why it is worth +familiarizing yourself with: + +- [Update instructions](../update/index.md) +- [Reference architectures](../administration/reference_architectures/index.md) +- [GitLab.com's architecture](https://about.gitlab.com/handbook/engineering/infrastructure/production/architecture/) +- [GitLab.com's upgrade pipeline](https://gitlab.com/gitlab-org/release/docs/blob/master/general/deploy/gitlab-com-deployer.md#upgrade-pipeline-default) + +To illustrate how these problems arise, take a look at this example: - 🚢 New version - 🙂 Old version diff --git a/doc/development/namespaces_storage_statistics.md b/doc/development/namespaces_storage_statistics.md index c6af0ddf9ea..e5263288210 100644 --- a/doc/development/namespaces_storage_statistics.md +++ b/doc/development/namespaces_storage_statistics.md @@ -53,7 +53,10 @@ SELECT split_part("rs".path, '/', 1) as root_path, COALESCE(SUM(ps.wiki_size), 0) AS wiki_size, COALESCE(SUM(ps.lfs_objects_size), 0) AS lfs_objects_size, COALESCE(SUM(ps.build_artifacts_size), 0) AS build_artifacts_size, - COALESCE(SUM(ps.packages_size), 0) AS packages_size + COALESCE(SUM(ps.pipeline_artifacts_size), 0) AS pipeline_artifacts_size, + COALESCE(SUM(ps.packages_size), 0) AS packages_size, + COALESCE(SUM(ps.snippets_size), 0) AS snippets_size, + COALESCE(SUM(ps.uploads_size), 0) AS uploads_size FROM "projects" INNER JOIN routes rs ON rs.source_id = projects.id AND rs.source_type = 'Project' INNER JOIN project_statistics ps ON ps.project_id = projects.id @@ -83,7 +86,10 @@ WITH refresh AS ( COALESCE(SUM(ps.wiki_size), 0) AS wiki_size, COALESCE(SUM(ps.lfs_objects_size), 0) AS lfs_objects_size, COALESCE(SUM(ps.build_artifacts_size), 0) AS build_artifacts_size, - COALESCE(SUM(ps.packages_size), 0) AS packages_size + COALESCE(SUM(ps.pipeline_artifacts_size), 0) AS pipeline_artifacts_size, + COALESCE(SUM(ps.packages_size), 0) AS packages_size, + COALESCE(SUM(ps.snippets_size), 0) AS snippets_size, + COALESCE(SUM(ps.uploads_size), 0) AS uploads_size FROM "projects" INNER JOIN routes rs ON rs.source_id = projects.id AND rs.source_type = 'Project' INNER JOIN project_statistics ps ON ps.project_id = projects.id @@ -94,7 +100,10 @@ SET storage_size = refresh.storage_size, wiki_size = refresh.wiki_size, lfs_objects_size = refresh.lfs_objects_size, build_artifacts_size = refresh.build_artifacts_size, - packages_size = refresh.packages_size + pipeline_artifacts_size = refresh.pipeline_artifacts_size, + packages_size = refresh.packages_size, + snippets_size = refresh.snippets_size, + uploads_size = refresh.uploads_size FROM refresh INNER JOIN routes rs ON rs.path = refresh.root_path AND rs.source_type = 'Namespace' WHERE namespace_storage_statistics.namespace_id = rs.source_id diff --git a/doc/development/pipelines.md b/doc/development/pipelines.md index 45982d6075b..71a11d2024c 100644 --- a/doc/development/pipelines.md +++ b/doc/development/pipelines.md @@ -23,6 +23,30 @@ as much as possible. After a merge request has been approved, the pipeline would contain the full RSpec & Jest tests. This will ensure that all tests have been run before a merge request is merged. +### Overview of the GitLab project test dependency + +To understand how the minimal test jobs are executed, we need to understand the dependency between +GitLab code (frontend and backend) and the respective tests (Jest and RSpec). +This dependency can be visualized in the following diagram: + +```mermaid +flowchart LR + subgraph frontend + fe["Frontend code"]--tested with-->jest + end + subgraph backend + be["Backend code"]--tested with-->rspec + end + + be--generates-->fixtures["frontend fixtures"] + fixtures--used in-->jest +``` + +In summary: + +- RSpec tests are dependent on the backend code. +- Jest tests are dependent on both frontend and backend code, the latter through the frontend fixtures. + ### RSpec minimal jobs #### Determining related RSpec test files in a merge request @@ -57,7 +81,7 @@ In this mode, `jest` would resolve all the dependencies of related to the change In addition, there are a few circumstances where we would always run the full Jest tests: -- when the `pipeline:run-all-rspec` label is set on the merge request +- when the `pipeline:run-all-jest` label is set on the merge request - when the merge request is created by an automation (e.g. Gitaly update or MR targeting a stable branch) - when any CI config file is changed (i.e. `.gitlab-ci.yml` or `.gitlab/ci/**/*`) - when any frontend "core" file is changed (i.e. `package.json`, `yarn.lock`, `babel.config.js`, `jest.config.*.js`, `config/helpers/**/*.js`) @@ -142,6 +166,13 @@ Our current RSpec tests parallelization setup is as follows: After that, the next pipeline uses the up-to-date `knapsack/report-master.json` file. +### Flaky tests + +Tests that are [known to be flaky](testing_guide/flaky_tests.md#automatic-retries-and-flaky-tests-detection) are: + +- skipped if the `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is set to `true` (`false` by default) +- run if `$SKIP_FLAKY_TESTS_AUTOMATICALLY` variable is not set to `true` or if the `~"pipeline:run-flaky-tests"` label is set on the MR + ### Monitoring The GitLab test suite is [monitored](performance.md#rspec-profiling) for the `main` branch, and any branch @@ -157,6 +188,8 @@ that includes `rspec-profile` in their name. Consult the [Review Apps](testing_guide/review_apps.md) dedicated page for more information. +If you want to force a Review App to be deployed regardless of your changes, you can add the `pipeline:run-review-app` label to the merge request. + ## As-if-FOSS jobs The `* as-if-foss` jobs run the GitLab test suite "as if FOSS", meaning as if the jobs would run in the context @@ -590,7 +623,8 @@ The current stages are: - `build-images`: This stage includes jobs that prepare Docker images that are needed by jobs in subsequent stages or downstream pipelines. - `fixtures`: This stage includes jobs that prepare fixtures needed by frontend tests. -- `test`: This stage includes most of the tests, DB/migration jobs, and static analysis jobs. +- `lint`: This stage includes linting and static analysis jobs. +- `test`: This stage includes most of the tests, and DB/migration jobs. - `post-test`: This stage includes jobs that build reports or gather data from the `test` stage's jobs (for example, coverage, Knapsack metadata, and so on). - `review-prepare`: This stage includes a job that build the CNG images that are @@ -664,7 +698,7 @@ then included in individual jobs via [`extends`](../ci/yaml/index.md#extends). The `rules` definitions are composed of `if:` conditions and `changes:` patterns, which are also defined in [`rules.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rules.gitlab-ci.yml) -and included in `rules` definitions via [YAML anchors](../ci/yaml/index.md#anchors) +and included in `rules` definitions via [YAML anchors](../ci/yaml/yaml_optimization.md#anchors) #### `if:` conditions @@ -696,7 +730,6 @@ and included in `rules` definitions via [YAML anchors](../ci/yaml/index.md#ancho | `if-dot-com-gitlab-org-and-security-merge-request` | Limit jobs creation to merge requests for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | | `if-dot-com-gitlab-org-and-security-tag` | Limit jobs creation to tags for the `gitlab-org` and `gitlab-org/security` groups on GitLab.com. | | | `if-dot-com-ee-schedule` | Limits jobs to scheduled pipelines for the `gitlab-org/gitlab` project on GitLab.com. | | -| `if-cache-credentials-schedule` | Limits jobs to scheduled pipelines with the `$CI_REPO_CACHE_CREDENTIALS` variable set. | | | `if-security-pipeline-merge-result` | Matches if the pipeline is for a security merge request triggered by `@gitlab-release-tools-bot`. | | <!-- vale gitlab.Substitutions = YES --> @@ -741,8 +774,10 @@ request, be sure to start the `dont-interrupt-me` job before pushing. [`.gitlab/ci/global.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/global.gitlab-ci.yml), with fixed keys: - `.setup-test-env-cache` + - `.ruby-cache` - `.rails-cache` - `.static-analysis-cache` + - `.rubocop-cache` - `.coverage-cache` - `.danger-review-cache` - `.qa-cache` @@ -752,7 +787,7 @@ request, be sure to start the `dont-interrupt-me` job before pushing. 1. Only the following jobs, running in 2-hourly scheduled pipelines, are pushing (that is, updating) to the caches: - `update-setup-test-env-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-gitaly-binaries-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - - `update-static-analysis-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). + - `update-rubocop-cache`, defined in [`.gitlab/ci/rails.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/rails.gitlab-ci.yml). - `update-qa-cache`, defined in [`.gitlab/ci/qa.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/qa.gitlab-ci.yml). - `update-assets-compile-production-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). - `update-assets-compile-test-cache`, defined in [`.gitlab/ci/frontend.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/frontend.gitlab-ci.yml). @@ -764,71 +799,28 @@ request, be sure to start the `dont-interrupt-me` job before pushing. We limit the artifacts that are saved and retrieved by jobs to the minimum in order to reduce the upload/download time and costs, as well as the artifacts storage. -### Pre-clone step - -The `gitlab-org/gitlab` project on GitLab.com uses a [pre-clone step](https://gitlab.com/gitlab-org/gitlab/-/issues/39134) -to seed the project with a recent archive of the repository. This is done for -several reasons: - -- It speeds up builds because a 800 MB download only takes seconds, as opposed to a full Git clone. -- It significantly reduces load on the file server, as smaller deltas mean less time spent in `git pack-objects`. - -The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable -[defined by GitLab.com shared runners](../ci/runners/build_cloud/linux_build_cloud.md#pre-clone-script). - -The `CI_PRE_CLONE_SCRIPT` is currently defined as a project CI/CD variable: - -```shell -( - echo "Downloading archived master..." - wget -O /tmp/gitlab.tar.gz https://storage.googleapis.com/gitlab-ci-git-repo-cache/project-278964/gitlab-master-shallow.tar.gz - - if [ ! -f /tmp/gitlab.tar.gz ]; then - echo "Repository cache not available, cloning a new directory..." - exit - fi - - rm -rf $CI_PROJECT_DIR - echo "Extracting tarball into $CI_PROJECT_DIR..." - mkdir -p $CI_PROJECT_DIR - cd $CI_PROJECT_DIR - tar xzf /tmp/gitlab.tar.gz - rm -f /tmp/gitlab.tar.gz - chmod a+w $CI_PROJECT_DIR -) -``` - -The first step of the script downloads `gitlab-master.tar.gz` from -Google Cloud Storage. There is a [GitLab CI job named `cache-repo`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/.gitlab/ci/cache-repo.gitlab-ci.yml#L5) -that is responsible for keeping that archive up-to-date. Every two hours -on a scheduled pipeline, it does the following: +### Git fetch caching -1. Creates a fresh clone of the `gitlab-org/gitlab` repository on GitLab.com. -1. Saves the data as a `.tar.gz`. -1. Uploads it into the Google Cloud Storage bucket. +Because GitLab.com uses the [pack-objects cache](../administration/gitaly/configure_gitaly.md#pack-objects-cache), +concurrent Git fetches of the same pipeline ref are deduplicated on +the Gitaly server (always) and served from cache (when available). -When a CI job runs with this configuration, the output looks something like this: +This works well for the following reasons: -```shell -$ eval "$CI_PRE_CLONE_SCRIPT" -Downloading archived master... -Extracting tarball into /builds/group/project... -Fetching changes... -Reinitialized existing Git repository in /builds/group/project/.git/ -``` +- The pack-objects cache is enabled on all Gitaly servers on GitLab.com. +- The CI/CD [Git strategy setting](../ci/pipelines/settings.md#choose-the-default-git-strategy) for `gitlab-org/gitlab` is **Git clone**, + causing all jobs to fetch the same data, which maximizes the cache hit ratio. +- We use [shallow clone](../ci/pipelines/settings.md#limit-the-number-of-changes-fetched-during-clone) to avoid downloading the full Git + history for every job. -Note that the `Reinitialized existing Git repository` message shows that -the pre-clone step worked. The runner runs `git init`, which -overwrites the Git configuration with the appropriate settings to fetch -from the GitLab repository. +### Pre-clone step -`CI_REPO_CACHE_CREDENTIALS` contains the Google Cloud service account -JSON for uploading to the `gitlab-ci-git-repo-cache` bucket. (If you're a -GitLab Team Member, find credentials in the -[GitLab shared 1Password account](https://about.gitlab.com/handbook/security/#1password-for-teams). +NOTE: +We no longer use this optimization for `gitlab-org/gitlab` because the [pack-objects cache](../administration/gitaly/configure_gitaly.md#pack-objects-cache) +allows Gitaly to serve the full CI/CD fetch traffic now. See [Git fetch caching](#git-fetch-caching). -Note that this bucket should be located in the same continent as the -runner, or [you can incur network egress charges](https://cloud.google.com/storage/pricing). +The pre-clone step works by using the `CI_PRE_CLONE_SCRIPT` variable +[defined by GitLab.com shared runners](../ci/runners/runner_cloud/linux_runner_cloud.md#pre-clone-script). --- diff --git a/doc/development/policies.md b/doc/development/policies.md index 8ad3d3f42ba..e5191f00d9a 100644 --- a/doc/development/policies.md +++ b/doc/development/policies.md @@ -111,7 +111,7 @@ Each line represents a rule that was evaluated. There are a few things to note: Here you can see that the first four rules were evaluated `false` for which user and subject. For example, you can see in the last line that -the rule was activated because the user `john` had Reporter access to +the rule was activated because the user `john` had the Reporter [role](../user/permissions.md) on `Project/4`. When a policy is asked whether a particular ability is allowed diff --git a/doc/development/reactive_caching.md b/doc/development/reactive_caching.md index 3c0a1419604..5aaf4c72e64 100644 --- a/doc/development/reactive_caching.md +++ b/doc/development/reactive_caching.md @@ -74,7 +74,7 @@ For more information, read the internal issue ### In models and integrations -The ReactiveCaching concern can be used in models as well as `integrations` +The ReactiveCaching concern can be used in models as well as integrations (`app/models/integrations`). 1. Include the concern in your model or integration. @@ -88,7 +88,7 @@ The ReactiveCaching concern can be used in models as well as `integrations` To include the concern in an integration: ```ruby - include ReactiveService + include Integrations::ReactivelyCached ``` 1. Implement the `calculate_reactive_cache` method in your model or integration. @@ -201,15 +201,15 @@ There are some `class_attribute` options which can be tweaked. and `"ExampleModel:1:arg1:arg2:alive"` respectively, where `ExampleModel` is the name of the model, `1` is the ID of the record, `arg1` and `arg2` are parameters passed to `with_reactive_cache`. -- If you're including this concern in a service instead, you must override - the default by adding the following to your service: +- If you're including this concern in an integration (`app/models/integrations/`) instead, you must override + the default by adding the following to your integration: ```ruby - self.reactive_cache_key = ->(service) { [service.class.model_name.singular, service.project_id] } + self.reactive_cache_key = ->(integration) { [integration.class.model_name.singular, integration.project_id] } ``` If your reactive_cache_key is exactly like the above, you can use the existing - `ReactiveService` concern instead. + `Integrations::ReactivelyCached` concern instead. #### `self.reactive_cache_lease_timeout` diff --git a/doc/development/redis.md b/doc/development/redis.md index fa07cebdc61..75170b8c746 100644 --- a/doc/development/redis.md +++ b/doc/development/redis.md @@ -21,8 +21,7 @@ GitLab uses [Redis](https://redis.io) for the following distinct purposes: In most environments (including the GDK), all of these point to the same Redis instance. -On GitLab.com, we use [separate Redis -instances](../administration/redis/replication_and_failover.md#running-multiple-redis-clusters). +On GitLab.com, we use [separate Redis instances](../administration/redis/replication_and_failover.md#running-multiple-redis-clusters). See the [Redis SRE guide](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/redis/redis-survival-guide-for-sres.md) for more details on our setup. @@ -121,8 +120,7 @@ on GitLab.com On GitLab.com, entries from the [Redis slow log](https://redis.io/commands/slowlog) are available in the -`pubsub-redis-inf-gprd*` index with the [`redis.slowlog` -tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time_s),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). +`pubsub-redis-inf-gprd*` index with the [`redis.slowlog` tag](https://log.gprd.gitlab.net/app/kibana#/discover?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-1d,to:now))&_a=(columns:!(json.type,json.command,json.exec_time_s),filters:!(('$state':(store:appState),meta:(alias:!n,disabled:!f,index:AWSQX_Vf93rHTYrsexmk,key:json.tag,negate:!f,params:(query:redis.slowlog),type:phrase),query:(match:(json.tag:(query:redis.slowlog,type:phrase))))),index:AWSQX_Vf93rHTYrsexmk)). This shows commands that have taken a long time and may be a performance concern. diff --git a/doc/development/reusing_abstractions.md b/doc/development/reusing_abstractions.md index 568e8a9d123..ccf82dc6c77 100644 --- a/doc/development/reusing_abstractions.md +++ b/doc/development/reusing_abstractions.md @@ -133,8 +133,20 @@ Everything in `lib/api`. Everything that resides in `app/services`. +Services should consider inheriting from: + +- `BaseContainerService` for services scoped by container (project or group) +- `BaseProjectService` for services scoped to projects +- `BaseGroupService` for services scoped to groups + +or, create a new base class and update the list above. + +Legacy classes inherited from `BaseService` for historical reasons. + In Service classes the use of `execute` and `#execute` is preferred over `call` and `#call`. +Classes that are not service objects should be [created elsewhere](directory_structure.md#use-namespaces-to-define-bounded-contexts), such as in `lib`. + #### ServiceResponse Service classes usually have an `execute` method, which can return a diff --git a/doc/development/ruby3_gotchas.md b/doc/development/ruby3_gotchas.md new file mode 100644 index 00000000000..e4ed5039e3c --- /dev/null +++ b/doc/development/ruby3_gotchas.md @@ -0,0 +1,140 @@ +--- +stage: none +group: unassigned +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Ruby 3 gotchas + +This section documents several problems we found while working on [Ruby 3 support](https://gitlab.com/groups/gitlab-org/-/epics/5149) +and which led to subtle bugs or test failures that were difficult to understand. We encourage every GitLab contributor +who writes Ruby code on a regular basis to familiarize themselves with these issues. + +To find the complete list of changes to the Ruby 3 language and standard library, see +[Ruby Changes](https://rubyreferences.github.io/rubychanges/3.0.html). + +## `Hash#each` consistently yields a 2-element array to lambdas + +Consider the following code snippet: + +```ruby +def foo(a, b) + p [a, b] +end + +def bar(a, b = 2) + p [a, b] +end + +foo_lambda = method(:foo).to_proc +bar_lambda = method(:bar).to_proc + +{ a: 1 }.each(&foo_lambda) +{ a: 1 }.each(&bar_lambda) +``` + +In Ruby 2.7, the output of this program suggests that yielding hash entries to lambdas behaves +differently depending on how many required arguments there are: + +```ruby +# Ruby 2.7 +{ a: 1 }.each(&foo_lambda) # prints [:a, 1] +{ a: 1 }.each(&bar_lambda) # prints [[:a, 1], 2] +``` + +Ruby 3 makes this behavior consistent and always attempts to yield hash entries as a single `[key, value]` array: + +```ruby +# Ruby 3.0 +{ a: 1 }.each(&foo_lambda) # `foo': wrong number of arguments (given 1, expected 2) (ArgumentError) +{ a: 1 }.each(&bar_lambda) # prints [[:a, 1], 2] +``` + +To write code that works under both 2.7 and 3.0, consider the following options: + +- Always pass the lambda body as a block: `{ a: 1 }.each { |a, b| p [a, b] }`. +- Deconstruct the lambda arguments: `{ a: 1 }.each(&->((a, b)) { p [a, b] })`. + +We recommend always passing the block explicitly, and prefer two required arguments as block parameters. + +To learn more, see [Ruby issue 12706](https://bugs.ruby-lang.org/issues/12706). + +## `Symbol#to_proc` returns signature metadata consistent with lambdas + +A common idiom in Ruby is to obtain `Proc` objects using the `&:<symbol>` shorthand and +pass them to higher-order functions: + +```ruby +[1, 2, 3].each(&:to_s) +``` + +Ruby desugars `&:<symbol>` to `Symbol#to_proc`. We can call it with +the method _receiver_ as its first argument (here: `Integer`), and all method _arguments_ +(here: none) as its remaining arguments. + +This behaves the same in both Ruby 2.7 and Ruby 3. Where Ruby 3 diverges is when capturing +this `Proc` object and inspecting its call signature. +This is often done when writing DSLs or using other forms of meta-programming: + +```ruby +p = :foo.to_proc # This usually happens via a conversion through `&:foo` + +# Ruby 2.7: prints [[:rest]] (-1) +# Ruby 3.0: prints [[:req], [:rest]] (-2) +puts "#{p.parameters} (#{p.arity})" +``` + +Ruby 2.7 reports zero required and one optional parameter for this `Proc` object, while Ruby 3 reports one required +and one optional parameter. Ruby 2.7 is incorrect: the first argument must +always be passed, as it is the receiver of the method the `Proc` object represents, and methods cannot be +called without a receiver. + +Ruby 3 corrects this: the code that tests `Proc` object arity or parameter lists might now break and +has to be updated. + +To learn more, see [Ruby issue 16260](https://bugs.ruby-lang.org/issues/16260). + +## `OpenStruct` does not evaluate fields lazily + +The `OpenStruct` implementation has undergone a partial rewrite in Ruby 3, resulting in +behavioral changes. In Ruby 2.7, `OpenStruct` defines methods lazily, when the method is first accessed. +In Ruby 3.0, it defines these methods eagerly in the initializer, which can break classes that inherit from `OpenStruct` +and override these methods. + +Don't inherit from `OpenStruct` for these reasons; ideally, don't use it at all. +`OpenStruct` is [considered problematic](https://ruby-doc.org/stdlib-3.0.2/libdoc/ostruct/rdoc/OpenStruct.html#class-OpenStruct-label-Caveats). +When writing new code, prefer a `Struct` instead, which is simpler in implementation, although less flexible. + +## `Regexp` and `Range` instances are frozen + +It is not necessary anymore to explicitly freeze `Regexp` or `Range` instances because Ruby 3 freezes +them automatically upon creation. + +This has a subtle side-effect: Tests that stub method calls on these types now fail with an error because +RSpec cannot stub frozen objects: + +```ruby +# Ruby 2.7: works +# Ruby 3.0: error: "can't modify frozen object" +allow(subject.function_returning_range).to receive(:max).and_return(42) +``` + +Rewrite affected tests by not stubbing method calls on frozen objects. The example above can be rewritten as: + +```ruby +# Works with any Ruby version +allow(subject).to receive(:function_returning_range).and_return(1..42) +``` + +## Table tests fail with Ruby 3.0.2 + +Ruby 3.0.2 has a known bug that causes [table tests](testing_guide/best_practices.md#table-based--parameterized-tests) +to fail when table values consist of integer values. +The reasons are documented in [issue 337614](https://gitlab.com/gitlab-org/gitlab/-/issues/337614). +This problem has been fixed in Ruby and the fix is expected to be included in Ruby 3.0.3. + +The problem only affects users who run an unpatched Ruby 3.0.2. This is likely the case when you +installed Ruby manually or via tools like `asdf`. Users of the `gitlab-development-kit (GDK)` +are also affected by this problem. + +Build images are not affected because they include the patch set addressing this bug. diff --git a/doc/development/ruby_upgrade.md b/doc/development/ruby_upgrade.md index ad6bff8499a..f9816986b2d 100644 --- a/doc/development/ruby_upgrade.md +++ b/doc/development/ruby_upgrade.md @@ -137,7 +137,7 @@ This is typically necessary, since gems or Ruby applications that we maintain ou update these repositories for the GitLab Rails application to work with a new Ruby, it is good practice to keep Ruby versions in lock-step across all our repositories. For minor and major upgrades, add new CI/CD jobs to these repositories using the new Ruby. -A [build matrix definition](../ci/yaml/index.md#parallel-matrix-jobs) can do this efficiently. +A [build matrix definition](../ci/yaml/index.md#parallelmatrix) can do this efficiently. #### Decide which repositories to update diff --git a/doc/development/secure_coding_guidelines.md b/doc/development/secure_coding_guidelines.md index 0f13b8ecae9..65fdb815f87 100644 --- a/doc/development/secure_coding_guidelines.md +++ b/doc/development/secure_coding_guidelines.md @@ -540,6 +540,94 @@ out, _ = exec.Command("sh", "-c", "echo 1 | cat /etc/passwd").Output() This outputs `1` followed by the content of `/etc/passwd`. +## General recommendations + +### TLS minimum recommended version + +As we have [moved away from supporting TLS 1.0 and 1.1](https://about.gitlab.com/blog/2018/10/15/gitlab-to-deprecate-older-tls/), we should only use TLS 1.2 and above. + +#### Ciphers + +We recommend using the ciphers that Mozilla is providing in their [recommended SSL configuration generator](https://ssl-config.mozilla.org/#server=go&version=1.17&config=intermediate&guideline=5.6) for TLS 1.2: + +- `ECDHE-ECDSA-AES128-GCM-SHA256` +- `ECDHE-RSA-AES128-GCM-SHA256` +- `ECDHE-ECDSA-AES256-GCM-SHA384` +- `ECDHE-RSA-AES256-GCM-SHA384` +- `ECDHE-ECDSA-CHACHA20-POLY1305` +- `ECDHE-RSA-CHACHA20-POLY1305` + +And the following cipher suites (according to the [RFC 8446](https://datatracker.ietf.org/doc/html/rfc8446#appendix-B.4)) for TLS 1.3: + +- `TLS_AES_128_GCM_SHA256` +- `TLS_AES_256_GCM_SHA384` +- `TLS_CHACHA20_POLY1305_SHA256` + +*Note*: **Golang** does [not support](https://github.com/golang/go/blob/go1.17/src/crypto/tls/cipher_suites.go#L676) all cipher suites with TLS 1.3. + +##### Implementation examples + +##### TLS 1.3 + +For TLS 1.3, **Golang** only supports [3 cipher suites](https://github.com/golang/go/blob/go1.17/src/crypto/tls/cipher_suites.go#L676), as such we only need to set the TLS version: + +```golang +cfg := &tls.Config{ + MinVersion: tls.VersionTLS13, +} +``` + +For **Ruby**, you can use [HTTParty](https://github.com/jnunemaker/httparty) and specify TLS 1.3 version as well as ciphers: + +Whenever possible this example should be **avoided** for security purposes: + +```ruby +response = HTTParty.get('https://gitlab.com', ssl_version: :TLSv1_3, ciphers: ['TLS_AES_128_GCM_SHA256', 'TLS_AES_256_GCM_SHA384', 'TLS_CHACHA20_POLY1305_SHA256']) +``` + +When using [`GitLab::HTTP`](#gitlab-http-library), the code looks like: + +This is the **recommended** implementation to avoid security issues such as SSRF: + +```ruby +response = GitLab::HTTP.perform_request(Net::HTTP::Get, 'https://gitlab.com', ssl_version: :TLSv1_3, ciphers: ['TLS_AES_128_GCM_SHA256', 'TLS_AES_256_GCM_SHA384', 'TLS_CHACHA20_POLY1305_SHA256']) +``` + +##### TLS 1.2 + +**Golang** does support multiple cipher suites that we do not want to use with TLS 1.2. We need to explicitly list authorized ciphers: + +```golang +func secureCipherSuites() []uint16 { + return []uint16{ + tls.TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256, + tls.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256, + tls.TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, + tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, + tls.TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305, + tls.TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305, + } +``` + +And then use `secureCipherSuites()` in `tls.Config`: + +```golang +tls.Config{ + (...), + CipherSuites: secureCipherSuites(), + MinVersion: tls.VersionTLS12, + (...), +} +``` + +This example was taken [here](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/871b52dc700f1a66f6644fbb1e78a6d463a6ff83/internal/tool/tlstool/tlstool.go#L72). + +For **Ruby**, you can use again [HTTParty](https://github.com/jnunemaker/httparty) and specify this time TLS 1.2 version alongside with the recommended ciphers: + +```ruby +response = GitLab::HTTP.perform_request(Net::HTTP::Get, 'https://gitlab.com', ssl_version: :TLSv1_2, ciphers: ['ECDHE-ECDSA-AES128-GCM-SHA256', 'ECDHE-RSA-AES128-GCM-SHA256', 'ECDHE-ECDSA-AES256-GCM-SHA384', 'ECDHE-RSA-AES256-GCM-SHA384', 'ECDHE-ECDSA-CHACHA20-POLY1305', 'ECDHE-RSA-CHACHA20-POLY1305']) +``` + ## GitLab Internal Authorization ### Introduction @@ -592,3 +680,90 @@ In order to prevent this from happening, it is recommended to use the method `us forbidden!(api_access_denied_message(user)) end ``` + +## Guidelines when defining missing methods with metaprogramming + +Metaprogramming is a way to define methods **at runtime**, instead of at the time of writing and deploying the code. It is a powerful tool, but can be dangerous if we allow untrusted actors (like users) to define their own arbitrary methods. For example, imagine we accidentally let an attacker overwrite an access control method to always return true! It can lead to many classes of vulnerabilities such as access control bypass, information disclosure, arbitrary file reads, and remote code execution. + +Key methods to watch out for are `method_missing`, `define_method`, `delegate`, and similar methods. + +### Insecure metaprogramming example + +This example is adapted from an example submitted by [@jobert](https://hackerone.com/jobert?type=user) through our HackerOne bug bounty program. +Thank you for your contribution! + +Before Ruby 2.5.1, you could implement delegators using the `delegate` or `method_missing` methods. For example: + +```ruby +class User + def initialize(attributes) + @options = OpenStruct.new(attributes) + end + + def is_admin? + name.eql?("Sid") # Note - never do this! + end + + def method_missing(method, *args) + @options.send(method, *args) + end +end +``` + +When a method was called on a `User` instance that didn't exist, it passed it along to the `@options` instance variable. + +```ruby +User.new({name: "Jeeves"}).is_admin? +# => false + +User.new(name: "Sid").is_admin? +# => true + +User.new(name: "Jeeves", "is_admin?" => true).is_admin? +# => false +``` + +Because the `is_admin?` method is already defined on the class, its behavior is not overridden when passing `is_admin?` to the initializer. + +This class can be refactored to use the `Forwardable` method and `def_delegators`: + +```ruby +class User + extend Forwardable + + def initialize(attributes) + @options = OpenStruct.new(attributes) + + self.class.instance_eval do + def_delegators :@options, *attributes.keys + end + end + + def is_admin? + name.eql?("Sid") # Note - never do this! + end +end +``` + +It might seem like this example has the same behavior as the first code example. However, there's one crucial difference: **because the delegators are meta-programmed after the class is loaded, it can overwrite existing methods**: + +```ruby +User.new({name: "Jeeves"}).is_admin? +# => false + +User.new(name: "Sid").is_admin? +# => true + +User.new(name: "Jeeves", "is_admin?" => true).is_admin? +# => true +# ^------------------ The method is overwritten! Sneaky Jeeves! +``` + +In the example above, the `is_admin?` method is overwritten when passing it to the initializer. + +### Best practices + +- Never pass user-provided details into method-defining metaprogramming methods. + - If you must, be **very** confident that you've sanitized the values correctly. + Consider creating an allowlist of values, and validating the user input against that. +- When extending classes that use metaprogramming, make sure you don't inadvertently override any method definition safety checks. diff --git a/doc/development/service_ping/implement.md b/doc/development/service_ping/implement.md index 34a845147dc..65a8b4c1cad 100644 --- a/doc/development/service_ping/implement.md +++ b/doc/development/service_ping/implement.md @@ -295,19 +295,22 @@ Examples of implementation: - Using Redis methods [`INCR`](https://redis.io/commands/incr), [`GET`](https://redis.io/commands/get), and [`Gitlab::UsageDataCounters::WikiPageCounter`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/wiki_page_counter.rb) - Using Redis methods [`HINCRBY`](https://redis.io/commands/hincrby), [`HGETALL`](https://redis.io/commands/hgetall), and [`Gitlab::UsageCounters::PodLogs`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_counters/pod_logs.rb) -##### UsageData API tracking +##### `UsageData` API -<!-- There's nearly identical content in `##### Adding new events`. If you fix errors here, you may need to fix the same errors in the other location. --> +You can use the `UsageData` API to track events. +To track events, the `usage_data_api` feature flag must +be enabled (set to `default_enabled: true`). +Enabled by default in GitLab 13.7 and later. -1. Track event using `UsageData` API +##### UsageData API tracking - Increment event count using ordinary Redis counter, for given event name. +1. Track events using the [`UsageData` API](#usagedata-api). - Tracking events using the `UsageData` API requires the `usage_data_api` feature flag to be enabled, which is enabled by default. + Increment event count using an ordinary Redis counter, for a given event name. API requests are protected by checking for a valid CSRF token. - To be able to increment the values, the related feature `usage_data_<event_name>` should be enabled. + To increment the values, the related feature `usage_data_<event_name>` must be enabled. ```plaintext POST /usage_data/increment_counter @@ -315,18 +318,18 @@ Examples of implementation: | Attribute | Type | Required | Description | | :-------- | :--- | :------- | :---------- | - | `event` | string | yes | The event name it should be tracked | + | `event` | string | yes | The event name to track. | Response: - - `200` if event was tracked - - `400 Bad request` if event parameter is missing - - `401 Unauthorized` if user is not authenticated - - `403 Forbidden` for invalid CSRF token provided + - `200` if the event was tracked. + - `400 Bad request` if the event parameter is missing. + - `401 Unauthorized` if the user is not authenticated. + - `403 Forbidden` if an invalid CSRF token is provided. -1. Track events using JavaScript/Vue API helper which calls the API above +1. Track events using the JavaScript/Vue API helper which calls the [`UsageData` API](#usagedata-api). - Note that `usage_data_api` and `usage_data_#{event_name}` should be enabled to be able to track events + To track events, `usage_data_api` and `usage_data_#{event_name}` must be enabled. ```javascript import api from '~/api'; @@ -460,14 +463,10 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF track_usage_event(:incident_management_incident_created, current_user.id) ``` - - Using the `UsageData` API. - <!-- There's nearly identical content in `##### UsageData API Tracking`. If you find / fix errors here, you may need to fix errors in that section too. --> + - Using the [`UsageData` API](#usagedata-api). Increment unique users count using Redis HLL, for a given event name. - To track events using the `UsageData` API, ensure the `usage_data_api` feature flag - is set to `default_enabled: true`. Enabled by default in GitLab 13.7 and later. - API requests are protected by checking for a valid CSRF token. ```plaintext @@ -485,10 +484,7 @@ Implemented using Redis methods [PFADD](https://redis.io/commands/pfadd) and [PF - `401 Unauthorized` if the user is not authenticated. - `403 Forbidden` if an invalid CSRF token is provided. - - Using the JavaScript/Vue API helper, which calls the `UsageData` API. - - To track events using the `UsageData` API, ensure the `usage_data_api` feature flag - is set to `default_enabled: true`. Enabled by default in GitLab 13.7 and later. + - Using the JavaScript/Vue API helper, which calls the [`UsageData` API](#usagedata-api). Example for an existing event already defined in [known events](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/): @@ -559,13 +555,18 @@ We can also disable tracking completely by using the global flag: ##### Known events are added automatically in Service Data payload -All events added in [`known_events/common.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events/common.yml) are automatically added to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). +Service Ping adds all events [`known_events/*.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/known_events) to Service Data generation under the `redis_hll_counters` key. This column is stored in [version-app as a JSON](https://gitlab.com/gitlab-services/version-gitlab-com/-/blob/master/db/schema.rb#L209). For each event we add metrics for the weekly and monthly time frames, and totals for each where applicable: - `#{event_name}_weekly`: Data for 7 days for daily [aggregation](#add-new-events) events and data for the last complete week for weekly [aggregation](#add-new-events) events. - `#{event_name}_monthly`: Data for 28 days for daily [aggregation](#add-new-events) events and data for the last 4 complete weeks for weekly [aggregation](#add-new-events) events. -Redis HLL implementation calculates automatic total metrics, if there are more than one metric for the same category, aggregation, and Redis slot. +Redis HLL implementation calculates total metrics when both of these conditions are met: + +- The category is manually included in [CATEGORIES_FOR_TOTALS](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/usage_data_counters/hll_redis_counter.rb#L21). +- There is more than one metric for the same category, aggregation, and Redis slot. + +We add total unique counts for the weekly and monthly time frames where applicable: - `#{category}_total_unique_counts_weekly`: Total unique counts for events in the same category for the last 7 days or the last complete week, if events are in the same Redis slot and we have more than one metric. - `#{category}_total_unique_counts_monthly`: Total unique counts for events in same category for the last 28 days or the last 4 complete weeks, if events are in the same Redis slot and we have more than one metric. diff --git a/doc/development/service_ping/index.md b/doc/development/service_ping/index.md index 19bf7446da9..6ddbe2f9646 100644 --- a/doc/development/service_ping/index.md +++ b/doc/development/service_ping/index.md @@ -50,35 +50,57 @@ We use the following terminology to describe the Service Ping components: - **Metrics**: primarily made up of row counts for different tables in an instance's database. Each metric has a corresponding [metric definition](metrics_dictionary.md#metrics-definition-and-validation) in a YAML file. +- **MAU**: monthly active users. +- **WAU**: weekly active users. ### Why should we enable Service Ping? - The main purpose of Service Ping is to build a better GitLab. Data about how GitLab is used is collected to better understand feature/stage adoption and usage, which helps us understand how GitLab is adding value and helps our team better understand the reasons why people use GitLab and with this knowledge we're able to make better product decisions. - As a benefit of having Service Ping active, GitLab lets you analyze the users' activities over time of your GitLab installation. -- As a benefit of having Service Ping active, GitLab provides you with The DevOps Report,which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. +- As a benefit of having Service Ping active, GitLab provides you with [DevOps Score](../../user/admin_area/analytics/dev_ops_report.md#devops-score), which gives you an overview of your entire instance's adoption of Concurrent DevOps from planning to monitoring. - You get better, more proactive support. (assuming that our TAMs and support organization used the data to deliver more value) - You get insight and advice into how to get the most value out of your investment in GitLab. Wouldn't you want to know that a number of features or values are not being adopted in your organization? - You get a report that illustrates how you compare against other similar organizations (anonymized), with specific advice and recommendations on how to improve your DevOps processes. - Service Ping is enabled by default. To disable it, see [Disable Service Ping](#disable-service-ping). - When Service Ping is enabled, you have the option to participate in our [Registration Features Program](#registration-features-program) and receive free paid features. -#### Registration Features Program +### Limitations + +- Service Ping does not track frontend events things like page views, link clicks, or user sessions. +- Service Ping focuses only on aggregated backend events. + +Because of these limitations we recommend you: + +- Instrument your products with Snowplow for more detailed analytics on GitLab.com. +- Use Service Ping to track aggregated backend events on self-managed instances. + +### Registration Features Program > Introduced in GitLab 14.1. -Starting with GitLab version 14.1, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data via [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. +In GitLab versions 14.1 and later, free self-managed users running [GitLab EE](../ee_features.md) can receive paid features by registering with GitLab and sending us activity data through [Service Ping](#what-is-service-ping). Features introduced here do not remove the feature from its paid tier. Users can continue to access the features in a paid tier without sharing usage data. -##### Features available in 14.1 and later +#### Features available in 14.1 and later 1. [Email from GitLab](../../tools/email.md). +#### Features available in 14.4 and later + +1. [Repository size limit](../../user/admin_area/settings/account_and_limit_settings.md#repository-size-limit). +1. [Restrict group access by IP address](../../user/group/index.md#restrict-group-access-by-ip-address). + NOTE: Registration is not yet required for participation, but will be added in a future milestone. -### Limitations +#### Enable Registration Features -- Service Ping does not track frontend events things like page views, link clicks, or user sessions, and only focuses on aggregated backend events. -- Because of these limitations we recommend instrumenting your products with Snowplow for more detailed analytics on GitLab.com and use Service Ping to track aggregated backend events on self-managed. +1. Sign in as a user with the [Administrator](../../user/permissions.md) role. +1. On the top bar, select **Menu > Admin**. +1. On the left sidebar, select **Settings > Metrics and profiling**. +1. Expand the **Usage statistics** section. +1. If not enabled, select the **Enable Service Ping** checkbox. +1. Select the **Enable Registration Features** checkbox. +1. Select **Save changes**. ## View the Service Ping payload **(FREE SELF)** @@ -180,7 +202,7 @@ sequenceDiagram S3 Bucket->>Snowflake DW: Import data Snowflake DW->>Snowflake DW: Transform data using dbt Snowflake DW->>Sisense Dashboards: Data available for querying - Versions Application->>GitLab Instance: DevOps Report (Conversational Development Index) + Versions Application->>GitLab Instance: DevOps Score (Conversational Development Index) ``` ## How Service Ping works @@ -209,10 +231,6 @@ We also collect metrics specific to [Geo](../../administration/geo/index.md) sec "repositories_replication_enabled"=>true, "repositories_synced_count"=>24, "repositories_failed_count"=>0, - "attachments_replication_enabled"=>true, - "attachments_count"=>1, - "attachments_synced_count"=>1, - "attachments_failed_count"=>0, "git_fetch_event_count_weekly"=>nil, "git_push_event_count_weekly"=>nil, ... other geo node status fields @@ -450,27 +468,56 @@ bin/rake gitlab:usage_data:dump_sql_in_json bin/rake gitlab:usage_data:dump_sql_in_yaml > ~/Desktop/usage-metrics-2020-09-02.yaml ``` -## Generating and troubleshooting Service Ping - -This activity is to be done via a detached screen session on a remote server. +## Generate Service Ping -Before you begin these steps, make sure the key is added to the SSH agent locally -with the `ssh-add` command. +To generate Service Ping, use [Teleport](https://goteleport.com/docs/) or a detached screen session on a remote server. ### Triggering -1. Connect to bastion with agent forwarding: `$ ssh -A lb-bastion.gprd.gitlab.com` -1. Create named screen: `$ screen -S <username>_usage_ping_<date>` -1. Connect to console host: `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal` -1. Run `SubmitUsagePingService.new.execute` -1. Detach from screen: `ctrl + a, ctrl + d` -1. Exit from bastion: `$ exit` +#### Trigger Service Ping with Teleport + +1. Request temporary [access](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to the required environment. +1. After your approval is issued, [access the Rails console](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#access-approval). +1. Run `ServicePing::SubmitService.new.execute`. + +#### Trigger Service Ping with a detached screen session + +1. Connect to bastion with agent forwarding: + + `$ ssh -A lb-bastion.gprd.gitlab.com`. +1. Create named screen: + + `$ screen -S <username>_usage_ping_<date>`. +1. Connect to console host: + + `$ ssh $USER-rails@console-01-sv-gprd.c.gitlab-production.internal`. +1. Run `ServicePing::SubmitService.new.execute` +1. Detach from screen: + + `ctrl + a, ctrl + d` +1. Exit from bastion: + + `$ exit` ### Verification (After approx 30 hours) -1. Reconnect to bastion: `$ ssh -A lb-bastion.gprd.gitlab.com` -1. Find your screen session: `$ screen -ls` -1. Attach to your screen session: `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22` +#### Verify with a detached screen session + +1. Follow [the steps](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/Teleport/Connect_to_Rails_Console_via_Teleport.md#how-to-use-teleport-to-connect-to-rails-console) to request a new access to the required environment and connect to the Rails console +1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` +1. Check the when the payload was sent: `RawUsageData.last.sent_at` + +#### Verify using detached screen session + +1. Reconnect to bastion: + + `$ ssh -A lb-bastion.gprd.gitlab.com` +1. Find your screen session: + + `$ screen -ls` +1. Attach to your screen session: + + `$ screen -x 14226.mwawrzyniak_usage_ping_2021_01_22` 1. Check the last payload in `raw_usage_data` table: `RawUsageData.last.payload` 1. Check the when the payload was sent: `RawUsageData.last.sent_at` diff --git a/doc/development/service_ping/metrics_dictionary.md b/doc/development/service_ping/metrics_dictionary.md index c1478e6290e..b3c404de150 100644 --- a/doc/development/service_ping/metrics_dictionary.md +++ b/doc/development/service_ping/metrics_dictionary.md @@ -197,18 +197,28 @@ tier: The GitLab codebase provides a dedicated [generator](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/generators/gitlab/usage_metric_definition_generator.rb) to create new metric definitions. -For uniqueness, the generated file includes a timestamp prefix, in ISO 8601 format. +For uniqueness, the generated files include a timestamp prefix in ISO 8601 format. -The generator takes the key path argument and 2 options and creates the metric YAML definition in corresponding location: +The generator takes a list of key paths and 2 options as arguments. It creates metric YAML definitions in the corresponding location: - `--ee`, `--no-ee` Indicates if metric is for EE. -- `--dir=DIR` indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`. +- `--dir=DIR` Indicates the metric directory. It must be one of: `counts_7d`, `7d`, `counts_28d`, `28d`, `counts_all`, `all`, `settings`, `license`. + +**Single metric example** ```shell bundle exec rails generate gitlab:usage_metric_definition counts.issues --dir=7d create config/metrics/counts_7d/issues.yml ``` +**Multiple metrics example** + +```shell +bundle exec rails generate gitlab:usage_metric_definition counts.issues counts.users --dir=7d +create config/metrics/counts_7d/issues.yml +create config/metrics/counts_7d/users.yml +``` + NOTE: To create a metric definition used in EE, add the `--ee` flag. diff --git a/doc/development/sidekiq_style_guide.md b/doc/development/sidekiq_style_guide.md index d45e2073fe7..e28a328888d 100644 --- a/doc/development/sidekiq_style_guide.md +++ b/doc/development/sidekiq_style_guide.md @@ -255,6 +255,11 @@ module Ci end ``` +Also, you can pass `if_deduplicated: :reschedule_once` option to re-run a job once after +the currently running job finished and deduplication happened at least once. +This ensures that the latest result is always produced even if a race condition +happened. See [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/342123) for more information. + #### Scheduling jobs in the future GitLab doesn't skip jobs scheduled in the future, as we assume that @@ -279,6 +284,36 @@ module AuthorizedProjectUpdate end ``` +### Setting the deduplication time-to-live (TTL) + +Deduplication depends on an idempotency key that is stored in Redis. This is normally +cleared by the configured deduplication strategy. + +However, the key can remain until its TTL in certain cases like: + +1. `until_executing` is used but the job was never enqueued or executed after the Sidekiq + client middleware was run. + +1. `until_executed` is used but the job fails to finish due to retry exhaustion, gets + interrupted the maximum number of times, or gets lost. + +The default value is 6 hours. During this time, jobs won't be enqueued even if the first +job never executed or finished. + +The TTL can be configured with: + +```ruby +class ProjectImportScheduleWorker + include ApplicationWorker + + idempotent! + deduplicate :until_executing, ttl: 5.minutes +end +``` + +Duplicate jobs can happen when the TTL is reached, so make sure you lower this only for jobs +that can tolerate some duplication. + ### Deduplication with load balancing > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6763) in GitLab 14.4. @@ -740,8 +775,7 @@ We use the following approach to determine whether a worker is CPU-bound: ## Feature category -All Sidekiq workers must define a known [feature -category](feature_categorization/index.md#sidekiq-workers). +All Sidekiq workers must define a known [feature category](feature_categorization/index.md#sidekiq-workers). ## Job weights diff --git a/doc/development/snowplow/implementation.md b/doc/development/snowplow/implementation.md index 0d81b442850..fe1de789eae 100644 --- a/doc/development/snowplow/implementation.md +++ b/doc/development/snowplow/implementation.md @@ -102,14 +102,12 @@ track_action: "click_button" }) ### Implement Vue component tracking For custom event tracking, use a Vue `mixin` in components. Vue `mixin` exposes the `Tracking.event` -static method and the `track` method called from components or templates. You can specify tracking -options in `data` or `computed`. These options override any defaults and allow the values to be dynamic -from props or based on state. +static method and the `track` method. You can specify tracking options in `data` or `computed`. +These options override any defaults and allow the values to be dynamic from props or based on state. -Default options are passed when an event is tracked from the component. If you don't specify an option, -the default `document.body.dataset.page` is used. The default options are: +Several default options are passed when an event is tracked from the component: -- `category` +- `category`: If you don't specify, by default `document.body.dataset.page` is used. - `label` - `property` - `value` @@ -426,7 +424,7 @@ records the same events as the full Snowplow pipeline. To query events, use the To install and run Snowplow Micro, complete these steps to modify the [GitLab Development Kit (GDK)](https://gitlab.com/gitlab-org/gitlab-development-kit): -1. Ensure Docker is installed and running. +1. Ensure [Docker is installed](https://docs.docker.com/get-docker/) and running. 1. To install Snowplow Micro, clone the settings in [this project](https://gitlab.com/gitlab-org/snowplow-micro-configuration). @@ -438,73 +436,18 @@ To install and run Snowplow Micro, complete these steps to modify the ./snowplow-micro.sh ``` -1. Use GDK to start the PostgreSQL terminal and connect - to the `gitlabhq_development` database: +1. Set the environment variable to tell the GDK to use Snowplow Micro in development. This overrides two `application_settings` options: + - `snowplow_enabled` setting will instead return `true` from `Gitlab::Tracking.enabled?` + - `snowplow_collector_hostname` setting will instead always return `localhost:9090` (or whatever is set for `SNOWPLOW_MICRO_URI`) from `Gitlab::Tracking.collector_hostname`. ```shell - gdk psql -d gitlabhq_development + export SNOWPLOW_MICRO_ENABLE=1 ``` -1. Update your instance's settings to enable Snowplow events and - point to the Snowplow Micro collector: + Optionally, you can set the URI for you Snowplow Micro instance as well (the default value is `http://localhost:9090`): ```shell - update application_settings set snowplow_collector_hostname='localhost:9090', snowplow_enabled=true, snowplow_cookie_domain='.gitlab.com'; - ``` - -1. Update `DEFAULT_SNOWPLOW_OPTIONS` in `app/assets/javascripts/tracking/constants.js` to remove `forceSecureTracker: true`: - - ```diff - diff --git a/app/assets/javascripts/tracking/constants.js b/app/assets/javascripts/tracking/constants.js - index 598111e4086..eff38074d4c 100644 - --- a/app/assets/javascripts/tracking/constants.js - +++ b/app/assets/javascripts/tracking/constants.js - @@ -7,7 +7,6 @@ export const DEFAULT_SNOWPLOW_OPTIONS = { - appId: '', - userFingerprint: false, - respectDoNotTrack: true, - - forceSecureTracker: true, - eventMethod: 'post', - contexts: { webPage: true, performanceTiming: true }, - formTracking: false, - ``` - -1. Update `options` in `lib/gitlab/tracking.rb` to add `protocol` and `port`: - - ```diff - diff --git a/lib/gitlab/tracking.rb b/lib/gitlab/tracking.rb - index 618e359211b..e9084623c43 100644 - --- a/lib/gitlab/tracking.rb - +++ b/lib/gitlab/tracking.rb - @@ -41,7 +41,9 @@ def options(group) - cookie_domain: Gitlab::CurrentSettings.snowplow_cookie_domain, - app_id: Gitlab::CurrentSettings.snowplow_app_id, - form_tracking: additional_features, - - link_click_tracking: additional_features - + link_click_tracking: additional_features, - + protocol: 'http', - + port: 9090 - }.transform_keys! { |key| key.to_s.camelize(:lower).to_sym } - end - ``` - -1. Update `emitter` in `lib/gitlab/tracking/destinations/snowplow.rb` to change `protocol`: - - ```diff - diff --git a/lib/gitlab/tracking/destinations/snowplow.rb b/lib/gitlab/tracking/destinations/snowplow.rb - index 4fa844de325..5dd9d0eacfb 100644 - --- a/lib/gitlab/tracking/destinations/snowplow.rb - +++ b/lib/gitlab/tracking/destinations/snowplow.rb - @@ -40,7 +40,7 @@ def tracker - def emitter - SnowplowTracker::AsyncEmitter.new( - Gitlab::CurrentSettings.snowplow_collector_hostname, - - protocol: 'https' - + protocol: 'http' - ) - end - end - + export SNOWPLOW_MICRO_URI=https://127.0.0.1:8080 ``` 1. Restart GDK: diff --git a/doc/development/snowplow/index.md b/doc/development/snowplow/index.md index b8b35857adf..f4f41221699 100644 --- a/doc/development/snowplow/index.md +++ b/doc/development/snowplow/index.md @@ -6,10 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Snowplow -This page provides an overview of how Snowplow works and how to enable it. - -## What is Snowplow - Snowplow is an enterprise-grade marketing and Product Intelligence platform that tracks how users engage with our website and application. [Snowplow](https://snowplowanalytics.com) consists of several loosely-coupled sub-systems: @@ -23,12 +19,6 @@ Snowplow is an enterprise-grade marketing and Product Intelligence platform that  -### Useful links - -- [Snowplow data structure](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/) -- [Our Iglu schema registry](https://gitlab.com/gitlab-org/iglu) -- [List of events used in our codebase (Event Dictionary)](https://metrics.gitlab.com/snowplow.html) - ## Enable Snowplow tracking Tracking can be enabled at: @@ -165,6 +155,9 @@ Snowplow JavaScript adds [web-specific parameters](https://docs.snowplowanalytic ## Related topics +- [Snowplow data structure](https://docs.snowplowanalytics.com/docs/understanding-your-pipeline/canonical-event/) +- [Our Iglu schema registry](https://gitlab.com/gitlab-org/iglu) +- [List of events used in our codebase (Event Dictionary)](https://metrics.gitlab.com/snowplow.html) - [Product Intelligence Guide](https://about.gitlab.com/handbook/product/product-intelligence-guide/) - [Service Ping Guide](../service_ping/index.md) - [Product Intelligence Direction](https://about.gitlab.com/direction/product-intelligence/) diff --git a/doc/development/testing_guide/best_practices.md b/doc/development/testing_guide/best_practices.md index 52e89a10556..6a739d9e1a5 100644 --- a/doc/development/testing_guide/best_practices.md +++ b/doc/development/testing_guide/best_practices.md @@ -50,6 +50,23 @@ bundle exec guard When using spring and guard together, use `SPRING=1 bundle exec guard` instead to make use of spring. +### Eager loading the application code + +By default, the application code: + +- Isn't eagerly loaded in the `test` environment. +- Is eagerly loaded in CI/CD (when `ENV['CI'].present?`) to surface any potential loading issues. + +If you need to enable eager loading when executing tests, +use the `GITLAB_TEST_EAGER_LOAD` environment variable: + +```shell +GITLAB_TEST_EAGER_LOAD=1 bin/rspec spec/models/project_spec.rb +``` + +If your test depends on all the application code that is being loaded, add the `:eager_load` tag. +This ensures that the application code is eagerly loaded before the test execution. + ### Ruby warnings > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/47767) in GitLab 13.7. diff --git a/doc/development/testing_guide/end_to_end/beginners_guide.md b/doc/development/testing_guide/end_to_end/beginners_guide.md index 7370cc5771b..27a87d25170 100644 --- a/doc/development/testing_guide/end_to_end/beginners_guide.md +++ b/doc/development/testing_guide/end_to_end/beginners_guide.md @@ -350,7 +350,7 @@ GITLAB_PASSWORD=<GDK root password> bundle exec bin/qa Test::Instance::All http: Where `<test_file>` is: - `qa/specs/features/browser_ui/1_manage/login/log_in_spec.rb` when running the Login example. -- `qa/specs/features/browser_ui/2_plan/issues/create_issue_spec.rb` when running the Issue example. +- `qa/specs/features/browser_ui/2_plan/issue/create_issue_spec.rb` when running the Issue example. ## End-to-end test merge request template diff --git a/doc/development/testing_guide/end_to_end/feature_flags.md b/doc/development/testing_guide/end_to_end/feature_flags.md index 994ee3f253c..52212410cc6 100644 --- a/doc/development/testing_guide/end_to_end/feature_flags.md +++ b/doc/development/testing_guide/end_to_end/feature_flags.md @@ -79,11 +79,21 @@ for details. End-to-end tests should pass with a feature flag enabled before it is enabled on Staging or on GitLab.com. Tests that need to be updated should be identified as part of [quad-planning](https://about.gitlab.com/handbook/engineering/quality/quad-planning/). The relevant [counterpart Software Engineer in Test](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors) is responsible for updating the tests or assisting another engineer to do so. However, if a change does not go through quad-planning and a required test update is not made, test failures could block deployment. -If a test enables a feature flag as describe above, it is sufficient to run the `package-and-qa` job in a merge request containing the relevant changes. -Or, if the feature flag and relevant changes have already been merged, you can confirm that the tests -pass on `main`. The end-to-end tests run on `main` every two hours, and the results are posted to a [Test -Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amaster). +### Automatic test execution when a feature flag definition changes -If the relevant tests do not enable the feature flag themselves, you can check if the tests will need -to be updated by opening a draft merge request that enables the flag by default and then running the `package-and-qa` job. +If a merge request adds or edits a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation), +two `package-and-qa` jobs will be included automatically in the merge request pipeline. One job will enable the defined +feature flag and the other will disable it. The jobs execute the same suite of tests to confirm that they pass with if +the feature flag is either enabled or disabled. + +### Test execution during feature development + +If an end-to-end test enables a feature flag, the end-to-end test suite can be used to test changes in a merge request +by running the `package-and-qa` job in the merge request pipeline. If the feature flag and relevant changes have already been merged, you can confirm that the tests +pass on the default branch. The end-to-end tests run on the default branch every two hours, and the results are posted to a [Test +Session Report, which is available in the testcase-sessions project](https://gitlab.com/gitlab-org/quality/testcase-sessions/-/issues?label_name%5B%5D=found%3Amain). + +If the relevant tests do not enable the feature flag themselves, you can check if the tests will need to be updated by opening +a draft merge request that enables the flag by default via a [feature flag definition file](../../feature_flags/index.md#feature-flag-definition-and-validation). +That will [automatically execute the end-to-end test suite](#automatic-test-execution-when-a-feature-flag-definition-changes). The merge request can be closed once the tests pass. If you need assistance to update the tests, please contact the relevant [stable counterpart in the Quality department](https://about.gitlab.com/handbook/engineering/quality/#individual-contributors), or any Software Engineer in Test if there is no stable counterpart for your group. diff --git a/doc/development/testing_guide/end_to_end/index.md b/doc/development/testing_guide/end_to_end/index.md index b097d6b0729..57c8c3bf59c 100644 --- a/doc/development/testing_guide/end_to_end/index.md +++ b/doc/development/testing_guide/end_to_end/index.md @@ -147,7 +147,7 @@ as well as these: | Variable | Description | |-|-| | `QA_SCENARIO` | The scenario to run (default `Test::Instance::Image`) | -| `QA_TESTS` | The test(s) to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, e.g., `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. | +| `QA_TESTS` | The test(s) to run (no default, which means run all the tests in the scenario). Use file paths as you would when running tests via RSpec, for example, `qa/specs/features/ee/browser_ui` would include all the `EE` UI tests. | | `QA_RSPEC_TAGS` | The RSpec tags to add (no default) | For now [manual jobs with custom variables don't use the same variable diff --git a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md index eadd0ef49a0..95984a701e7 100644 --- a/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md +++ b/doc/development/testing_guide/end_to_end/running_tests_that_require_special_setup.md @@ -165,20 +165,20 @@ QA_DEBUG=true WEBDRIVER_HEADLESS=false GITLAB_ADMIN_USERNAME=rootusername GITLAB The following includes more information on the command: --`QA_DEBUG` - Set to `true` to verbosely log page object actions. --`WEBDRIVER_HEADLESS` - When running locally, set to `false` to allow browser tests to be visible - watch your tests being run. --`GITLAB_ADMIN_USERNAME` - Administrator username to use when adding a license. --`GITLAB_ADMIN_PASSWORD` - Administrator password to use when adding a license. --`GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with administrator access. Used for API access as an administrator during tests. --`CLUSTER_API_URL` - Use the address `https://kubernetes.docker.internal:6443` . This address is used to enable the cluster to be network accessible while deploying using Auto DevOps. --`https://[YOUR-PORT].qa-tunnel.gitlab.info/` - The address of your local GDK --`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - The path to the monitor core specs --`--tag` - the meta-tags used to filter the specs correctly +- `QA_DEBUG` - Set to `true` to verbosely log page object actions. +- `WEBDRIVER_HEADLESS` - When running locally, set to `false` to allow browser tests to be visible - watch your tests being run. +- `GITLAB_ADMIN_USERNAME` - Administrator username to use when adding a license. +- `GITLAB_ADMIN_PASSWORD` - Administrator password to use when adding a license. +- `GITLAB_QA_ACCESS_TOKEN` and `GITLAB_QA_ADMIN_ACCESS_TOKEN` - A valid personal access token with the `api` scope. This is used for API access during tests, and is used in the version that staging is currently running. The `ADMIN_ACCESS_TOKEN` is from a user with administrator access. Used for API access as an administrator during tests. +- `CLUSTER_API_URL` - Use the address `https://kubernetes.docker.internal:6443` . This address is used to enable the cluster to be network accessible while deploying using Auto DevOps. +- `https://[YOUR-PORT].qa-tunnel.gitlab.info/` - The address of your local GDK +- `qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - The path to the monitor core specs +- `--tag` - the meta-tags used to filter the specs correctly At the moment of this writing, there are two specs which run monitor tests: --`qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - has the specs of features in GitLab Free --`qa/specs/features/ee/browser_ui/8_monitor/all_monitor_features_spec.rb` - has the specs of features for paid GitLab (Enterprise Edition) +- `qa/specs/features/browser_ui/8_monitor/all_monitor_core_features_spec.rb` - has the specs of features in GitLab Free +- `qa/specs/features/ee/browser_ui/8_monitor/all_monitor_features_spec.rb` - has the specs of features for paid GitLab (Enterprise Edition) ### How to debug @@ -485,3 +485,109 @@ To run the LDAP tests on your local with TLS disabled, follow these steps: ```shell GITLAB_LDAP_USERNAME="tanuki" GITLAB_LDAP_PASSWORD="password" QA_DEBUG=true WEBDRIVER_HEADLESS=false bin/qa Test::Instance::All http://localhost qa/specs/features/browser_ui/1_manage/login/log_into_gitlab_via_ldap_spec.rb ``` + +## Guide to the mobile suite + +### What are mobile tests + +Tests that are tagged with `:mobile` can be run against specified mobile devices using cloud emulator/simulator services. + +### How to run mobile tests with Sauce Labs + +Running directly against an environment like staging is not recommended because Sauce Labs test logs expose credentials. Therefore, it is best practice and the default to use a tunnel. + +Tunnel installation instructions are here [https://docs.saucelabs.com/secure-connections/sauce-connect/installation]. To start the tunnel, after following the installation above, copy the run command in Sauce Labs > Tunnels (must be logged in to Sauce Labs with the credentials found in 1Password) and run in terminal. + +NOTE: +It is highly recommended to use `GITLAB_QA_ACCESS_TOKEN` to speed up tests and reduce flakiness. + +`QA_REMOTE_MOBILE_DEVICE_NAME` can be any device name listed in [https://saucelabs.com/platform/supported-browsers-devices] under Emulators/simulators and the latest versions of Android or iOS. `QA_BROWSER` must be set to `safari` for iOS devices and `chrome` for Android devices. + +1. To test against a local instance with a tunnel running, in `gitlab/qa` run: + +```shell +$ QA_BROWSER="safari" \ + QA_REMOTE_MOBILE_DEVICE_NAME="iPhone 12 Simulator" \ + QA_REMOTE_GRID="ondemand.saucelabs.com:80" \ + QA_REMOTE_GRID_USERNAME="gitlab-sl" \ + QA_REMOTE_GRID_ACCESS_KEY="<found in Sauce Lab account>" \ + GITLAB_QA_ACCESS_TOKEN="<token>" \ + bundle exec bin/qa Test::Instance::All http://<local_ip>:3000 -- <relative_spec_path> +``` + +Results can be watched in real time while logged into Sauce Labs under AUTOMATED > Test Results. + +### How to add an existing test to the mobile suite + +The main reason a test might fail when adding the `:mobile` tag is navigation differences in desktop vs mobile layouts, therefore the test needs to be updated to use mobile navigation when running mobile tests. + +If an existing method needs to be changed or a new one created, a new mobile page object should be created in `qa/qa/mobile/page/` and it should be prepended in the original page object by adding: + +```ruby +prepend Mobile::Page::NewPageObject if Runtime::Env.mobile_layout? +``` + +For example to change an existing method when running mobile tests: + +New mobile page object: + +```ruby +module QA + module Mobile + module Page + module Project + module Show + extend QA::Page::PageConcern + + def self.prepended(base) + super + + base.class_eval do + prepend QA::Mobile::Page::Main::Menu + + view 'app/assets/javascripts/nav/components/top_nav_new_dropdown.vue' do + element :new_issue_mobile_button + end + end + end + + def go_to_new_issue + open_mobile_new_dropdown + + click_element(:new_issue_mobile_button) + end + end + end + end + end +end +``` + +Original page object prepending the new mobile if there's a mobile layout: + +```ruby +module QA + module Page + module Project + class Show < Page::Base + prepend Mobile::Page::Project::Show if Runtime::Env.mobile_layout? + + view 'app/views/layouts/header/_new_dropdown.html.haml' do + element :new_menu_toggle + end + + view 'app/helpers/nav/new_dropdown_helper.rb' do + element :new_issue_link + end + + def go_to_new_issue + click_element(:new_menu_toggle) + click_element(:new_issue_link) + end + end + end + end +end +``` + +When running mobile tests for phone layouts, both `remote_mobile_device_name` and `mobile_layout` are `true` but when using a tablet layout, only `remote_mobile_device_name` is true. This is because phone layouts have more menus closed by default such as how both tablets and phones have the left nav closed but unlike phone layouts, tablets have the regular top navigation bar, not the mobile one. So in the case where the navigation being edited needs to be used in tablet layouts as well, use `remote_mobile_device_name` instead of `mobile_layout?` when prepending so it will use it if it's a tablet layout as well. diff --git a/doc/development/testing_guide/end_to_end/style_guide.md b/doc/development/testing_guide/end_to_end/style_guide.md index 22ddae6e836..a9571df1188 100644 --- a/doc/development/testing_guide/end_to_end/style_guide.md +++ b/doc/development/testing_guide/end_to_end/style_guide.md @@ -14,7 +14,7 @@ This document describes the conventions used at GitLab for writing End-to-end (E When clicking in a single link to navigate, use `click_`. -E.g.: +For example: ```ruby def click_ci_cd_pipelines @@ -33,7 +33,7 @@ From a testing perspective, if we want to check that clicking a link, or a butto When interacting with multiple elements to go to a page, use `go_to_`. -E.g.: +For example: ```ruby def go_to_operations_environments @@ -63,12 +63,12 @@ We follow a simple formula roughly based on Hungarian notation. - `type`: A generic control on the page that can be seen by a user. - `_button` - `_checkbox` - - `_container`: an element that includes other elements, but doesn't present visible content itself. E.g., an element that has a third-party editor inside it, but which isn't the editor itself and so doesn't include the editor's content. + - `_container`: an element that includes other elements, but doesn't present visible content itself. For example, an element that has a third-party editor inside it, but which isn't the editor itself and so doesn't include the editor's content. - `_content`: any element that contains text, images, or any other content displayed to the user. - `_dropdown` - `_field`: a text input element. - `_link` - - `_modal`: a popup modal dialog, e.g., a confirmation prompt. + - `_modal`: a popup modal dialog, for example, a confirmation prompt. - `_placeholder`: a temporary element that appears while content is loading. For example, the elements that are displayed instead of discussions while the discussions are being fetched. - `_radio` - `_tab` @@ -116,7 +116,7 @@ we use the name of the page object in [snake_case](https://en.wikipedia.org/wiki (all lowercase, with words separated by an underscore). See good and bad examples below. While we prefer to follow the standard in most cases, it is also acceptable to -use common abbreviations (e.g., `mr`) or other alternatives, as long as +use common abbreviations (for example, `mr`) or other alternatives, as long as the name is not ambiguous. This can include appending `_page` if it helps to avoid confusion or make the code more readable. For example, if a page object is named `New`, it could be confusing to name the block argument `new` because that diff --git a/doc/development/testing_guide/frontend_testing.md b/doc/development/testing_guide/frontend_testing.md index 0e721ba2760..3096386d7c3 100644 --- a/doc/development/testing_guide/frontend_testing.md +++ b/doc/development/testing_guide/frontend_testing.md @@ -423,7 +423,8 @@ it('passes', () => { ### Waiting in tests Sometimes a test needs to wait for something to happen in the application before it continues. -Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/WindowOrWorkerGlobalScope/setTimeout) +Avoid using [`setTimeout`](https://developer.mozilla.org/en-US/docs/Web/API/setTimeout) + because it makes the reason for waiting unclear. Instead use one of the following approaches. #### Promises and Ajax calls diff --git a/doc/development/testing_guide/review_apps.md b/doc/development/testing_guide/review_apps.md index 4091f213a8f..3190ab6c899 100644 --- a/doc/development/testing_guide/review_apps.md +++ b/doc/development/testing_guide/review_apps.md @@ -16,6 +16,7 @@ For any of the following scenarios, the `start-review-app-pipeline` job would be - for merge requests with frontend changes - for merge requests with QA changes - for scheduled pipelines +- the MR has the `pipeline:run-review-app` label set ## QA runs on Review Apps @@ -37,6 +38,15 @@ browser performance testing using a ## How to +### Redeploy Review App from a clean slate + +To reset Review App and redeploy from a clean slate, do the following: + +1. Run `review-stop` job. +1. Re-deploy by running or retrying `review-deploy` job. + +Doing this will remove all existing data from a previously deployed Review App. + ### Get access to the GCP Review Apps cluster You need to [open an access request (internal link)](https://gitlab.com/gitlab-com/access-requests/-/issues/new) @@ -180,7 +190,7 @@ subgraph "CNG-mirror pipeline" **Additional notes:** - If the `review-deploy` job keeps failing (and a manual retry didn't help), - please post a message in the `#g_qe_engineering_productivity` channel and/or create a `~"Engineering Productivity"` `~"ep::review apps"` `~bug` + please post a message in the `#g_qe_engineering_productivity` channel and/or create a `~"Engineering Productivity"` `~"ep::review apps"` `~"type::bug"` issue with a link to your merge request. Note that the deployment failure can reveal an actual problem introduced in your merge request (i.e. this isn't necessarily a transient failure)! @@ -189,7 +199,7 @@ subgraph "CNG-mirror pipeline" your merge request. You can also download the artifacts to see screenshots of the page at the time the failures occurred. If you don't find the cause of the failure or if it seems unrelated to your change, please post a message in the - `#quality` channel and/or create a ~Quality ~bug issue with a link to your + `#quality` channel and/or create a ~Quality ~"type::bug" issue with a link to your merge request. - The manual `review-stop` can be used to stop a Review App manually, and is also started by GitLab once a merge @@ -238,6 +248,16 @@ Leading indicators may be health check failures leading to restarts or majority The [Review Apps Overview dashboard](https://console.cloud.google.com/monitoring/classic/dashboards/6798952013815386466?project=gitlab-review-apps&timeDomain=1d) aids in identifying load spikes on the cluster, and if nodes are problematic or the entire cluster is trending towards unhealthy. +### Database related errors in `review-deploy` or `review-qa-smoke` + +Occasionally the state of a Review App's database could diverge from the database schema. This could be caused by +changes to migration files or schema, such as a migration being renamed or deleted. This typically manifest in migration errors such as: + +- migration job failing with a column that already exists +- migration job failing with a column that does not exist + +To recover from this, please attempt to [redeploy Review App from a clean slate](#redeploy-review-app-from-a-clean-slate) + ### Release failed with `ImagePullBackOff` **Potential cause:** diff --git a/doc/development/testing_guide/testing_migrations_guide.md b/doc/development/testing_guide/testing_migrations_guide.md index 757a70fb4e0..4092c1a2f6d 100644 --- a/doc/development/testing_guide/testing_migrations_guide.md +++ b/doc/development/testing_guide/testing_migrations_guide.md @@ -38,7 +38,7 @@ ensures proper isolation. ## Testing an `ActiveRecord::Migration` class -To test an `ActiveRecord::Migration` class (i.e., a +To test an `ActiveRecord::Migration` class (for example, a regular migration `db/migrate` or a post-migration `db/post_migrate`), you must load the migration file by using the `require_migration!` helper method because it is not autoloaded by Rails. @@ -61,14 +61,35 @@ Since the migration files are not autoloaded by Rails, you must manually load the migration file. To do so, you can use the `require_migration!` helper method which can automatically load the correct migration file based on the spec filename. -For example, if your spec file is named as `populate_foo_column_spec.rb` then the -helper method tries to load `${schema_version}_populate_foo_column.rb` migration file. +In GitLab 14.4 and later, you can use `require_migration!` to load migration files from spec files +that contain the schema version in the filename (for example, +`2021101412150000_populate_foo_column_spec.rb`). -In case there is no pattern between your spec file and the actual migration file, -you can provide the migration filename without the schema version, like so: +```ruby +# frozen_string_literal: true + +require 'spec_helper' +require_migration! + +RSpec.describe PopulateFooColumn do + ... +end +``` + +In some cases, you must require multiple migration files to use them in your specs. Here, there's no +pattern between your spec file and the other migration file. You can provide the migration filename +like so: ```ruby -require_migration!('populate_foo_column') +# frozen_string_literal: true + +require 'spec_helper' +require_migration! +require_migration!('populate_bar_column') + +RSpec.describe PopulateFooColumn do + ... +end ``` #### `table` @@ -213,54 +234,65 @@ Migration tests depend on what the migration does exactly, the most common types #### Example of a data migration test This spec tests the -[`db/post_migrate/20170526185842_migrate_pipeline_stages.rb`](https://gitlab.com/gitlab-org/gitlab-foss/blob/v11.6.5/db/post_migrate/20170526185842_migrate_pipeline_stages.rb) +[`db/post_migrate/20200723040950_migrate_incident_issues_to_incident_type.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/db/post_migrate/20200723040950_migrate_incident_issues_to_incident_type.rb) migration. You can find the complete spec in -[`spec/migrations/migrate_pipeline_stages_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/blob/v11.6.5/spec/migrations/migrate_pipeline_stages_spec.rb). +[`spec/migrations/migrate_incident_issues_to_incident_type_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/migrations/migrate_incident_issues_to_incident_type_spec.rb). ```ruby -require 'spec_helper' +# frozen_string_literal: true +require 'spec_helper' require_migration! -RSpec.describe MigratePipelineStages do - # Create test data - pipeline and CI/CD jobs. - let(:jobs) { table(:ci_builds) } - let(:stages) { table(:ci_stages) } - let(:pipelines) { table(:ci_pipelines) } +RSpec.describe MigrateIncidentIssuesToIncidentType do + let(:migration) { described_class.new } + let(:projects) { table(:projects) } + let(:namespaces) { table(:namespaces) } + let(:labels) { table(:labels) } + let(:issues) { table(:issues) } + let(:label_links) { table(:label_links) } + let(:label_props) { IncidentManagement::CreateIncidentLabelService::LABEL_PROPERTIES } + + let(:namespace) { namespaces.create!(name: 'foo', path: 'foo') } + let!(:project) { projects.create!(namespace_id: namespace.id) } + let(:label) { labels.create!(project_id: project.id, **label_props) } + let!(:incident_issue) { issues.create!(project_id: project.id) } + let!(:other_issue) { issues.create!(project_id: project.id) } + + # Issue issue_type enum + let(:issue_type) { 0 } + let(:incident_type) { 1 } before do - projects.create!(id: 123, name: 'gitlab1', path: 'gitlab1') - pipelines.create!(id: 1, project_id: 123, ref: 'master', sha: 'adf43c3a') - jobs.create!(id: 1, commit_id: 1, project_id: 123, stage_idx: 2, stage: 'build') - jobs.create!(id: 2, commit_id: 1, project_id: 123, stage_idx: 1, stage: 'test') + label_links.create!(target_id: incident_issue.id, label_id: label.id, target_type: 'Issue') end - # Test just the up migration. - it 'correctly migrates pipeline stages' do - expect(stages.count).to be_zero - - migrate! + describe '#up' do + it 'updates the incident issue type' do + expect { migrate! } + .to change { incident_issue.reload.issue_type } + .from(issue_type) + .to(incident_type) - expect(stages.count).to eq 2 - expect(stages.all.pluck(:name)).to match_array %w[test build] + expect(other_issue.reload.issue_type).to eql(issue_type) + end end - # Test a reversible migration. - it 'correctly migrates up and down pipeline stages' do - reversible_migration do |migration| - # Expectations will run before the up migration, - # and then again after the down migration - migration.before -> { - expect(stages.count).to be_zero - } - - # Expectations will run after the up migration. - migration.after -> { - expect(stages.count).to eq 2 - expect(stages.all.pluck(:name)).to match_array %w[test build] - } + describe '#down' do + let!(:incident_issue) { issues.create!(project_id: project.id, issue_type: issue_type) } + + it 'updates the incident issue type' do + migration.up + + expect { migration.down } + .to change { incident_issue.reload.issue_type } + .from(incident_type) + .to(issue_type) + + expect(other_issue.reload.issue_type).to eql(issue_type) end + end end ``` @@ -336,41 +368,62 @@ end ### Example background migration test This spec tests the -[`lib/gitlab/background_migration/archive_legacy_traces.rb`](https://gitlab.com/gitlab-org/gitlab-foss/blob/v11.6.5/lib/gitlab/background_migration/archive_legacy_traces.rb) +[`lib/gitlab/background_migration/backfill_draft_status_on_merge_requests.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/background_migration/backfill_draft_status_on_merge_requests.rb) background migration. You can find the complete spec on -[`spec/lib/gitlab/background_migration/archive_legacy_traces_spec.rb`](https://gitlab.com/gitlab-org/gitlab-foss/blob/v11.6.5/spec/lib/gitlab/background_migration/archive_legacy_traces_spec.rb) +[`spec/lib/gitlab/background_migration/backfill_draft_status_on_merge_requests_spec.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/spec/lib/gitlab/background_migration/backfill_draft_status_on_merge_requests_spec.rb) ```ruby +# frozen_string_literal: true + require 'spec_helper' -describe Gitlab::BackgroundMigration::ArchiveLegacyTraces, schema: 20180529152628 do - include TraceHelpers +RSpec.describe Gitlab::BackgroundMigration::BackfillDraftStatusOnMergeRequests do + let(:namespaces) { table(:namespaces) } + let(:projects) { table(:projects) } + let(:merge_requests) { table(:merge_requests) } - let(:namespaces) { table(:namespaces) } - let(:projects) { table(:projects) } - let(:builds) { table(:ci_builds) } - let(:job_artifacts) { table(:ci_job_artifacts) } + let(:group) { namespaces.create!(name: 'gitlab', path: 'gitlab') } + let(:project) { projects.create!(namespace_id: group.id) } - before do - namespaces.create!(id: 123, name: 'gitlab1', path: 'gitlab1') - projects.create!(id: 123, name: 'gitlab1', path: 'gitlab1', namespace_id: 123) - @build = builds.create!(id: 1, project_id: 123, status: 'success', type: 'Ci::Build') + let(:draft_prefixes) { ["[Draft]", "(Draft)", "Draft:", "Draft", "[WIP]", "WIP:", "WIP"] } + + def create_merge_request(params) + common_params = { + target_project_id: project.id, + target_branch: 'feature1', + source_branch: 'master' + } + + merge_requests.create!(common_params.merge(params)) end - context 'when trace file exists at the right place' do + context "for MRs with #draft? == true titles but draft attribute false" do + let(:mr_ids) { merge_requests.all.collect(&:id) } + before do - create_legacy_trace(@build, 'trace in file') + draft_prefixes.each do |prefix| + (1..4).each do |n| + create_merge_request( + title: "#{prefix} This is a title", + draft: false, + state_id: n + ) + end + end end - it 'correctly archive legacy traces' do - expect(job_artifacts.count).to eq(0) - expect(File.exist?(legacy_trace_path(@build))).to be_truthy + it "updates all open draft merge request's draft field to true" do + mr_count = merge_requests.all.count + + expect { subject.perform(mr_ids.first, mr_ids.last) } + .to change { MergeRequest.where(draft: false).count } + .from(mr_count).to(mr_count - draft_prefixes.length) + end - described_class.new.perform(1, 1) + it "marks successful slices as completed" do + expect(subject).to receive(:mark_job_as_succeeded).with(mr_ids.first, mr_ids.last) - expect(job_artifacts.count).to eq(1) - expect(File.exist?(legacy_trace_path(@build))).to be_falsy - expect(File.read(archived_trace_path(job_artifacts.first))).to eq('trace in file') + subject.perform(mr_ids.first, mr_ids.last) end end end diff --git a/doc/development/understanding_explain_plans.md b/doc/development/understanding_explain_plans.md index c3dfeaa6b92..e06ece38135 100644 --- a/doc/development/understanding_explain_plans.md +++ b/doc/development/understanding_explain_plans.md @@ -172,7 +172,7 @@ cost describing how expensive the entire node was. In general: the greater the values, the more expensive the node. When using `EXPLAIN ANALYZE`, these statistics will also include the actual time -(in milliseconds) spent, and other runtime statistics (e.g. the actual number of +(in milliseconds) spent, and other runtime statistics (for example, the actual number of produced rows): ```sql @@ -555,7 +555,7 @@ There are two terms commonly used for databases: cardinality, and selectivity. Cardinality refers to the number of unique values in a particular column in a table. -Selectivity is the number of unique values produced by an operation (e.g. an +Selectivity is the number of unique values produced by an operation (for example, an index scan or filter), relative to the total number of rows. The higher the selectivity, the more likely PostgreSQL is able to use an index. @@ -675,8 +675,8 @@ best avoided in most cases, such as: - Sequential scans on large tables - Filters that remove a lot of rows -- Performing a certain step (e.g. an index scan) that requires _a lot_ of - buffers (e.g. more than 512 MB for GitLab.com). +- Performing a certain step that requires _a lot_ of + buffers (for example, an index scan for GitLab.com that requires more than 512 MB). As a general guideline, aim for a query that: diff --git a/doc/development/uploads.md b/doc/development/uploads.md index 9c5686db0f6..6d8b951be83 100644 --- a/doc/development/uploads.md +++ b/doc/development/uploads.md @@ -7,12 +7,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Uploads development documentation [GitLab Workhorse](https://gitlab.com/gitlab-org/gitlab-workhorse) has special rules for handling uploads. -To prevent occupying a Ruby process on I/O operations, we process the upload in workhorse, where is cheaper. +We process the upload in Workhorse to prevent occupying a Ruby process on I/O operations and because it is cheaper. This process can also directly upload to object storage. ## The problem description -The following graph explains machine boundaries in a scalable GitLab installation. Without any workhorse optimization in place, we can expect incoming requests to follow the numbers on the arrows. +The following graph explains machine boundaries in a scalable GitLab installation. Without any Workhorse optimization in place, we can expect incoming requests to follow the numbers on the arrows. ```mermaid graph TB @@ -27,10 +27,10 @@ graph TB subgraph "redis cluster" r(persisted redis) end - LB-- 1 -->workhorse + LB-- 1 -->Workhorse subgraph "web or API fleet" - workhorse-- 2 -->rails + Workhorse-- 2 -->rails end rails-- "3 (write files)" -->nfs rails-- "4 (schedule a job)" -->r @@ -63,12 +63,12 @@ graph TB subgraph "redis cluster" r(persisted redis) end - LB-- 1 -->workhorse + LB-- 1 -->Workhorse subgraph "web or API fleet" - workhorse-- "3 (without files)" -->rails + Workhorse-- "3 (without files)" -->rails end - workhorse -- "2 (write files)" -->nfs + Workhorse -- "2 (write files)" -->nfs rails-- "4 (schedule a job)" -->r subgraph sidekiq @@ -120,7 +120,7 @@ We have three kinds of file encoding in our uploads: 1. <i class="fa fa-check-circle"></i> **multipart**: `multipart/form-data` is the most common, a file is encoded as a part of a multipart encoded request. 1. <i class="fa fa-check-circle"></i> **body**: some APIs uploads files as the whole request body. -1. <i class="fa fa-times-circle"></i> **JSON**: some JSON API uploads files as base64 encoded strings. This requires a change to GitLab Workhorse, +1. <i class="fa fa-times-circle"></i> **JSON**: some JSON APIs upload files as base64-encoded strings. This requires a change to GitLab Workhorse, which is tracked [in this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/325068). ## Uploading technologies @@ -131,9 +131,9 @@ GitLab supports 3 kinds of uploading technologies, here follows a brief descript ### Rack Multipart upload -This is the default kind of upload, and it's most expensive in terms of resources. +This is the default kind of upload, and it's the most expensive in terms of resources. -In this case, workhorse is unaware of files being uploaded and acts as a regular proxy. +In this case, Workhorse is unaware of files being uploaded and acts as a regular proxy. When a multipart request reaches the rails application, `Rack::Multipart` leaves behind temporary files in `/tmp` and uses valuable Ruby process time to copy files around. @@ -213,7 +213,7 @@ sequenceDiagram This is the more advanced acceleration technique we have in place. -Workhorse asks rails for temporary pre-signed object storage URLs and directly uploads to object storage. +Workhorse asks Rails for temporary pre-signed object storage URLs and directly uploads to object storage. In this setup, an extra Rails route must be implemented in order to handle authorization. Examples of this can be found in: @@ -221,7 +221,7 @@ In this setup, an extra Rails route must be implemented in order to handle autho and [its routes](https://gitlab.com/gitlab-org/gitlab/-/blob/cc723071ad337573e0360a879cbf99bc4fb7adb9/config/routes/git_http.rb#L31-32). - [API endpoints for uploading packages](packages.md#file-uploads). -This falls back to _disk buffered upload_ when `direct_upload` is disabled inside the [object storage setting](../administration/uploads.md#object-storage-settings). +Direct upload falls back to _disk buffered upload_ when `direct_upload` is disabled inside the [object storage setting](../administration/uploads.md#object-storage-settings). The answer to the `/authorize` call contains only a file system path. ```mermaid @@ -275,7 +275,7 @@ sequenceDiagram In this section, we describe how to add a new upload route [accelerated](#uploading-technologies) by Workhorse for [body and multipart](#upload-encodings) encoded uploads. -Uploads routes belong to one of these categories: +Upload routes belong to one of these categories: 1. Rails controllers: uploads handled by Rails controllers. 1. Grape API: uploads handled by a Grape API endpoint. @@ -289,7 +289,7 @@ GraphQL uploads do not support [direct upload](#direct-upload) yet. Depending on For both the Rails controller and Grape API uploads, Workhorse has to be updated in order to get the support for the new upload route. -1. Open an new issue in the [Workhorse tracker](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/new) describing precisely the new upload route: +1. Open a new issue in the [Workhorse tracker](https://gitlab.com/gitlab-org/gitlab-workhorse/-/issues/new) describing precisely the new upload route: - The route's URL. - The [upload encoding](#upload-encodings). - If possible, provide a dump of the upload request. diff --git a/doc/development/verifying_database_capabilities.md b/doc/development/verifying_database_capabilities.md index c5e854701c2..bda9c68eae5 100644 --- a/doc/development/verifying_database_capabilities.md +++ b/doc/development/verifying_database_capabilities.md @@ -12,13 +12,13 @@ necessary to add database (version) specific behavior. To facilitate this we have the following methods that you can use: -- `Gitlab::Database.main.version`: returns the PostgreSQL version number as a string +- `ApplicationRecord.database.version`: returns the PostgreSQL version number as a string in the format `X.Y.Z`. This allows you to write code such as: ```ruby -if Gitlab::Database.main.version.to_f >= 11.7 +if ApplicationRecord.database.version.to_f >= 11.7 run_really_fast_query else run_fast_query diff --git a/doc/development/workspaces/index.md b/doc/development/workspaces/index.md new file mode 100644 index 00000000000..b36c79e84d7 --- /dev/null +++ b/doc/development/workspaces/index.md @@ -0,0 +1,120 @@ +--- +comments: false +type: index, dev +stage: none +group: Development +info: "See the Technical Writers assigned to Development Guidelines: https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments-to-development-guidelines" +description: "Development Guidelines: learn about workspaces when developing GitLab." +--- + +# Workspaces + +The [Workspaces initiative](../../user/workspace/index.md) focuses on reaching feature parity between +SaaS and self-managed installations. + +## Consolidate groups and projects + +- [Architecture blueprint](../../architecture/blueprints/consolidating_groups_and_projects/index.md) + +One facet of the workspaces initiative is to consolidate groups and projects, +addressing the feature disparity between them. Some features, such as epics, are +only available at the group level. Some features, such as issues, are only available +at the project level. Other features, such as milestones, are available to both groups +and projects. + +We receive many requests to add features either to the group or project level. +Moving features around to different levels is problematic on multiple levels: + +- It requires engineering time to move the features. +- It requires UX overhead to maintain mental models of feature availability. +- It creates redundant code. + +When features are copied from one level (project, group, or instance) to another, +the copies often have small, nuanced differences between them. These nuances cause +extra engineering time when fixes are needed, because the fix must be copied to +several locations. These nuances also create different user experiences when the +feature is used in different places. + +A solution for this problem is to consolidate groups and projects into a single +entity, `namespace`. The work on this solution is split into several phases and +is tracked in [epic 6473](https://gitlab.com/groups/gitlab-org/-/epics/6473). + +### Phase 1 + +- [Phase 1 epic](https://gitlab.com/groups/gitlab-org/-/epics/6697). +- **Goals**: + 1. Ensure every project receives a corresponding record in the `namespaces` + table with `type='Project'`. + 1. For user namespaces, the type changes from `NULL` to `User`. + +We should make sure that projects, and the project namespace, are equivalent: + +- **Create project:** use Rails callbacks to ensure a new project namespace is + created for each project. Project namespace records should contain `created_at` and + `updated_at` attributes equal to the project's `created_at`/`updated_at` attributes. +- **Update project:** use the `after_save` callback in Rails to ensure some + attributes are kept in sync between project and project namespaces. + Read [`project#after_save`](https://gitlab.com/gitlab-org/gitlab/blob/6d26634e864d7b748dda0e283eb2477362263bc3/app/models/project.rb#L101-L101) + for more information. +- **Delete project:** use FKs cascade delete or Rails callbacks to ensure when a `Project` + or its `ProjectNamespace` is removed, its corresponding `ProjectNamespace` or `Project` + is also removed. +- **Transfer project to a different group:** make sure that when a project is transferred, + its corresponding project namespace is transferred to the same group. +- **Transfer group:** make sure when transferring a group that all of its sub-projects, + either direct or through descendant groups, have their corresponding project + namespaces transferred correctly as well. +- **Export or import project** + - **Export project** continues to export only the project, and not its project namespace, + in this phase. The project namespace does not contain any specific information + to export at this point. Eventually we want the project namespace to be exported as well. + - **Import project** creates a new project, so the project namespace is created through + Rails `after_save` callback on the project model. +- **Export or import group:** when importing or exporting a `Group`, projects are not + included in the operation. If that feature is changed to include `Project` when its group is + imported or exported, the logic must include their corresponding project namespaces + in the import or export. + +After ensuring these points, run a database migration to create a `ProjectNamespace` +record for every `Project`. Project namespace records created during the migration +should have `created_at` and `updated_at` attributes set to the migration runtime. +The project namespaces' `created_at` and `updated_at` attributes would not match +their corresponding project's `created_at` and `updated_at` attributes. We want +the different dates to help audit any of the created project namespaces, in case we need it. +After this work completes, we must migrate data as described in +[Backfill `ProjectNamespace` for every Project](https://gitlab.com/gitlab-org/gitlab/-/issues/337100). + +### Phase 2 + +- [Phase 2 epic](https://gitlab.com/groups/gitlab-org/-/epics/6768). +- **Goal**: Make `ProjectNamespace` the front entity to interact with instead of `Project`. + +In this phase: + +- Communicate the changes company-wide at the engineering level. We want to make + engineers aware of the upcoming changes, even though teams are not expected to + collaborate actively until phase 3. +- Raise awareness to avoid regressions, and conflicting or duplicate work that + can be dealt with before phase 3. + +Problems to solve as part of this phase: + +- Routes handling through `ProjectNamespace` rather than `Project`. +- Memberships handling. +- Policies handling. +- Import and export. +- Other interactions between project namespace and project models. + +### Phase 3 + +- [Phase 3 epic](https://gitlab.com/groups/gitlab-org/-/epics/6585). +- **Goal**: Feature parity between the namespace types. + +Phase 3 is when the active migration of features from `Project` to `ProjectNamespace`, +or directly to `Namespace`, happens. + +## Related topics + +- [Consolidating groups and projects](../../architecture/blueprints/consolidating_groups_and_projects/index.md) + architecture documentation +- [Workspace user documentation](../../user/workspace/index.md) diff --git a/doc/gitlab-basics/feature_branch_workflow.md b/doc/gitlab-basics/feature_branch_workflow.md index 99d346f2b01..989a1557ef2 100644 --- a/doc/gitlab-basics/feature_branch_workflow.md +++ b/doc/gitlab-basics/feature_branch_workflow.md @@ -16,7 +16,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/workflow.html' 1. Create branch with your feature: ```shell - git checkout -b $feature_name + git checkout -b feature_name ``` 1. Write code. Commit changes: @@ -28,7 +28,7 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/workflow.html' 1. Push your branch to GitLab: ```shell - git push origin $feature_name + git push origin feature_name ``` 1. Review your code on commits page. diff --git a/doc/index.md b/doc/index.md index a638647e3e3..57eb9e195ff 100644 --- a/doc/index.md +++ b/doc/index.md @@ -6,12 +6,18 @@ comments: false description: 'Learn how to use and administer GitLab, the most scalable Git-based fully integrated platform for software development.' --- +<!-- markdownlint-disable MD044 --> +<!-- MD044/proper-names test disabled after this line to make page compatible with markdownlint-cli 0.29.0. --> +<!-- See https://docs.gitlab.com/ee/development/documentation/testing.html#disable-markdownlint-tests --> + <div class="d-none"> <h3>Visit <a href="https://docs.gitlab.com/ee/">docs.gitlab.com</a> for the latest version of this help information with enhanced navigation, discoverability, and readability.</h3> </div> <!-- the div above will not display on the docs site but will display on /help --> +<!-- markdownlint-enable MD044 --> + # GitLab Docs Welcome to [GitLab](https://about.gitlab.com/) documentation. diff --git a/doc/install/aws/eks_clusters_aws.md b/doc/install/aws/eks_clusters_aws.md index 3c19a83f128..86318467a91 100644 --- a/doc/install/aws/eks_clusters_aws.md +++ b/doc/install/aws/eks_clusters_aws.md @@ -1,7 +1,6 @@ --- -type: reference, concepts stage: Enablement -group: Alliances +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- diff --git a/doc/install/aws/gitlab_hybrid_on_aws.md b/doc/install/aws/gitlab_hybrid_on_aws.md index 4d22a29ad0a..3c035b33be8 100644 --- a/doc/install/aws/gitlab_hybrid_on_aws.md +++ b/doc/install/aws/gitlab_hybrid_on_aws.md @@ -1,7 +1,6 @@ --- -type: reference, concepts stage: Enablement -group: Alliances +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- @@ -9,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Provision GitLab Cloud Native Hybrid on AWS EKS **(FREE SELF)** -GitLab "Cloud Native Hybrid" is a hybrid of the cloud native technology Kubernetes (EKS) and EC2. While as much of the GitLab application as possible runs in Kubernetes or on AWS services (PaaS), the GitLab service Gitaly must still be run on Ec2. Gitaly is a layer designed to overcome limitations of the Git binaries in a horizontally scaled architecture. You can read more here about why Gitaly was built and why the limitations of Git mean that it must currently run on instance compute in [Git Characteristics That Make Horizontal Scaling Difficult](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-characteristics-that-make-horizontal-scaling-difficult) +GitLab "Cloud Native Hybrid" is a hybrid of the cloud native technology Kubernetes (EKS) and EC2. While as much of the GitLab application as possible runs in Kubernetes or on AWS services (PaaS), the GitLab service Gitaly must still be run on Ec2. Gitaly is a layer designed to overcome limitations of the Git binaries in a horizontally scaled architecture. You can read more here about why Gitaly was built and why the limitations of Git mean that it must currently run on instance compute in [Git Characteristics That Make Horizontal Scaling Difficult](https://gitlab.com/gitlab-org/gitaly/-/blob/master/doc/DESIGN.md#git-characteristics-that-make-horizontal-scaling-difficult). Amazon provides a managed Kubernetes service offering known as [Amazon Elastic Kubernetes Service (EKS)](https://aws.amazon.com/eks/). @@ -18,10 +17,10 @@ Amazon provides a managed Kubernetes service offering known as [Amazon Elastic K | GitLab Cloud Native Hybrid Ref Arch | GitLab Baseline Perf Test Results Omnibus on Instances | AWS Bill of Materials (BOM) for CNH | AWS Build Performance Testing Results for [CNH](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | CNH Cost Estimate 3 AZs* | | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------------ | | [2K Omnibus](../../administration/reference_architectures/2k_users.md) | [2K Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/2k) | [2K Cloud Native Hybrid on EKS](#2k-cloud-native-hybrid-on-eks) | GPT Test Results | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=544bcf1162beae6b8130ad257d081cdf9d4504e3)<br />(2 AZ Cost Estimate is in BOM Below) | -| [3K](../../administration/reference_architectures/3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [3k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k) | [3K Cloud Native Hybrid on EKS](#3k-cloud-native-hybrid-on-eks) | [3K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt)<br /><br />[3K AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=f1294fec554e21be999711cddcdab9c5e7f83f14)<br />(2 AZ Cost Estimate is in BOM Below) | -| [5K](../../administration/reference_architectures/5k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [5k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k) | [5K Cloud Native Hybrid on EKS](#5k-cloud-native-hybrid-on-eks) | [5K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt)<br /><br />[5K AutoScale from 25% GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=330ee43c5b14662db5df6e52b34898d181a09e16) | -| [10K](../../administration/reference_architectures/10k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [10k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k) | [10K Cloud Native Hybrid on EKS](#10k-cloud-native-hybrid-on-eks) | [10K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt)<br /><br />[10K AutoScale GPT Test Results](hhttps://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | [10K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=5ac2e07a22e01c36ee76b5477c5a046cd1bea792) | -| [50K](../../administration/reference_architectures/50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [50k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k) | [50K Cloud Native Hybrid on EKS](#50k-cloud-native-hybrid-on-eks) | [50K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt)<br /><br />[10K AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | [50K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=b9c9d6ac1d4a7848011d2050cef3120931fb7c22) | +| [3K](../../administration/reference_architectures/3k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [3k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/3k) | [3K Cloud Native Hybrid on EKS](#3k-cloud-native-hybrid-on-eks) | [3K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt)<br /><br />[3K Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=f1294fec554e21be999711cddcdab9c5e7f83f14)<br />(2 AZ Cost Estimate is in BOM Below) | +| [5K](../../administration/reference_architectures/5k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [5k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/5k) | [5K Cloud Native Hybrid on EKS](#5k-cloud-native-hybrid-on-eks) | [5K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt)<br /><br />[5K AutoScale from 25% GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) | [1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=330ee43c5b14662db5df6e52b34898d181a09e16) | +| [10K](../../administration/reference_architectures/10k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [10k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/10k) | [10K Cloud Native Hybrid on EKS](#10k-cloud-native-hybrid-on-eks) | [10K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt)<br /><br />[10K Elastic Auto Scale GPT Test Results](hhttps://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | [10K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=5ac2e07a22e01c36ee76b5477c5a046cd1bea792) | +| [50K](../../administration/reference_architectures/50k_users.md#cloud-native-hybrid-reference-architecture-with-helm-charts-alternative) | [50k Baseline](https://gitlab.com/gitlab-org/quality/performance/-/wikis/Benchmarks/Latest/50k) | [50K Cloud Native Hybrid on EKS](#50k-cloud-native-hybrid-on-eks) | [50K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt)<br /><br />[10K Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | [50K 1 YR Ec2 Compute Savings + 1 YR RDS & Elasticache RIs](https://calculator.aws/#/estimate?id=b9c9d6ac1d4a7848011d2050cef3120931fb7c22) | \*Cost calculations for actual implementations are a rough guideline with the following considerations: @@ -36,7 +35,8 @@ Amazon provides a managed Kubernetes service offering known as [Amazon Elastic K The [AWS Quick Start for GitLab Cloud Native Hybrid on EKS](https://aws-quickstart.github.io/quickstart-eks-gitlab/) is developed by AWS, GitLab, and the community that contributes to AWS Quick Starts, whether directly to the GitLab Quick Start or to the underlying Quick Start dependencies GitLab inherits (for example, EKS Quick Start). NOTE: -This automation is in **Developer Preview**. GitLab is working with AWS on [these outstanding issues](https://github.com/aws-quickstart/quickstart-eks-gitlab/issues?q=is%3Aissue+is%3Aopen+%5BHL%5D) before it is fully released. +This automation is in **Developer Preview**. GitLab is working with AWS on resolving [the outstanding issues](https://github.com/aws-quickstart/quickstart-eks-gitlab/issues?q=is%3Aissue+is%3Aopen+%5BHL%5D) before it is fully released. You can subscribe to this issue to be notified of progress and release announcements: [AWS Quick Start for GitLab Cloud Native Hybrid on EKS Status: DEVELOPER PREVIEW](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues/11).<br><br> +The developer preview deploys Aurora PostgreSQL, but the release version will deploy Amazon RDS PostgreSQL due to [known issues](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues?label_name%5B%5D=AWS+Known+Issue) with Aurora. All performance testing results will also be redone after this change has been made. The [GitLab Environment Toolkit (GET)](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/tree/master) is an effort made by GitLab to create a multi-cloud, multi-GitLab (Omnibus + Cloud Native Hybrid) toolkit to provision GitLab. GET is developed by GitLab developers and is open to community contributions. It is helpful to review the [GitLab Environment Toolkit (GET) Issues](https://gitlab.com/gitlab-org/quality/gitlab-environment-toolkit/-/issues) to understand if any of them may affect your provisioning plans. @@ -79,7 +79,7 @@ The AWS Quick Start for GitLab Cloud Native Hybrid on EKS has been tested with G | **Request AWS Certificate Manager SSL certificate** | CreateSslCertificate | No | - The Quick Start creates public load balancer IPs, so that you can easily configure your local hosts file to get to the GUI for GitLab when deploying tests. However, you may need to manually alter this if public load balancers are not part of your provisioning plan. We are planning to make non-public load balancers a configuration option issue link: [Short Term: Documentation and/or Automation for private GitLab instance with no internet Ingress](https://github.com/aws-quickstart/quickstart-eks-gitlab/issues/55) -- As of 2021-08-19, AWS GovCloud has Graviton instances for Aurora PostgreSQL available, but does not for ElastiCache Redis. +- As of 2021-08-19, AWS GovCloud has Graviton instances for Amazon RDS PostgreSQL available, but does not for ElastiCache Redis. - It is challenging to get the Quick Start template to load in GovCloud from the Standard Quick Start URL, so the generic ones are provided here: - [Launch for New VPC in us-gov-east-1](https://us-gov-east-1.console.amazonaws-us-gov.com/cloudformation/home?region=us-gov-east-1#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-for-EKS-New-VPC) - [Launch for New VPC in us-gov-west-1](https://us-gov-west-1.console.amazonaws-us-gov.com/cloudformation/home?region=us-gov-west-1#/stacks/quickcreate?templateUrl=https://aws-quickstart.s3.us-east-1.amazonaws.com/quickstart-eks-gitlab/templates/gitlab-entry-new-vpc.template.yaml&stackName=Gitlab-for-EKS-New-VPC) @@ -95,7 +95,7 @@ Some services, such as log aggregation, outbound email are not specified by GitL | GitLab Services | AWS PaaS (Tested) | Provided by AWS Cloud <br />Native Hybrid Quick Start | | ------------------------------------------------------------ | ------------------------------ | ------------------------------------------------------------ | | <u>Tested PaaS Mentioned in Reference Architectures</u> | | | -| **PostgreSQL Database** | Aurora RDS | Yes. | +| **PostgreSQL Database** | Amazon RDS PostgreSQL | Yes. | | **Redis Caching** | Redis Elasticache | Yes. | | **Gitaly Cluster (Git Repository Storage)**<br />(Including Praefect and PostgreSQL) | ASG and Instances | Yes - ASG and Instances<br />**Note: Gitaly cannot be put into a Kubernetes Cluster.** | | **All GitLab storages besides Git Repository Storage**<br />(Includes Git-LFS which is S3 Compatible) | AWS S3 | Yes | @@ -126,9 +126,7 @@ Some services, such as log aggregation, outbound email are not specified by GitL **GPT Test Results** -- [3K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt) - -- [3K AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) +- TBD **Deploy Now** Deploy Now links leverage the AWS Quick Start automation and only prepopulate the number of instances and instance types for the Quick Start based on the Bill of Meterials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. @@ -158,17 +156,17 @@ On Demand pricing is used in this table for comparisons, but should not be used NOTE: If EKS node autoscaling is employed, it is likely that your average loading will run lower than this, especially during non-working hours and weekends. -| Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM<br />(Directly Usable in AWS Quick Start) | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | -| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------- | ------------------------------- | ---------------------------------------------------- | -| **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for perf. testing | | | -| **PostgreSQL**<br />AWS Aurora RDS Nodes Configuration (GPT tested) | 2vCPU, 7.5 GB<br />Tested with Graviton ARM | **db.r6g.large** x 3 nodes <br />(6vCPU, 48 GB) | 3 nodes x $0.26 = $0.78/hr | 3 nodes x $0.26 = $0.78/hr<br />(Aurora is always 3) | -| **Redis** | 1vCPU, 3.75GB<br />(across 12 nodes for Redis Cache, Redis Queues/Shared State, Sentinel Cache, Sentinel Queues/Shared State) | **cache.m6g.large** x 3 nodes<br />(6vCPU, 19GB) | 3 nodes x $0.15 = $0.45/hr | 2 nodes x $0.15 = $0.30/hr | -| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | | | | -| Gitaly Instances (in ASG) | 12 vCPU, 45GB<br />(across 3 nodes) | **m5.xlarge** x 3 nodes<br />(48 vCPU, 180 GB) | $0.192 x 3 = $0.58/hr | $0.192 x 3 = $0.58/hr | -| | The GitLab Reference architecture for 2K is not Highly Available and therefore has a single Gitaly no Praefect. AWS Quick Starts MUST be HA, so it implements Prafect from the 3K Ref Architecture to meet that requirement | | | | -| Praefect (Instances in ASG with load balancer) | 6 vCPU, 10 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **c5.large** x 3 nodes<br />(6 vCPU, 12 GB) | $0.09 x 3 = $0.21/hr | $0.09 x 3 = $0.21/hr | -| Praefect PostgreSQL(1) (AWS RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | N/A Reuses GitLab PostgreSQL | $0 | $0 | -| Internal Load Balancing Node | 2 vCPU, 1.8 GB | AWS ELB | $0.10/hr | $0.10/hr | +| Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM<br />(Directly Usable in AWS Quick Start) | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | +| ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------- | ------------------------------- | ------------------------------- | +| **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for perf. testing | | | +| **PostgreSQL**<br />AWS Amazon RDS PostgreSQL Nodes Configuration (GPT tested) | 2vCPU, 7.5 GB<br />Tested with Graviton ARM | **db.r6g.large** x 3 nodes <br />(6vCPU, 48 GB) | 3 nodes x $0.26 = $0.78/hr | 3 nodes x $0.26 = $0.78/hr | +| **Redis** | 1vCPU, 3.75GB<br />(across 12 nodes for Redis Cache, Redis Queues/Shared State, Sentinel Cache, Sentinel Queues/Shared State) | **cache.m6g.large** x 3 nodes<br />(6vCPU, 19GB) | 3 nodes x $0.15 = $0.45/hr | 2 nodes x $0.15 = $0.30/hr | +| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | | | | +| Gitaly Instances (in ASG) | 12 vCPU, 45GB<br />(across 3 nodes) | **m5.xlarge** x 3 nodes<br />(48 vCPU, 180 GB) | $0.192 x 3 = $0.58/hr | $0.192 x 3 = $0.58/hr | +| | The GitLab Reference architecture for 2K is not Highly Available and therefore has a single Gitaly no Praefect. AWS Quick Starts MUST be HA, so it implements Prafect from the 3K Ref Architecture to meet that requirement | | | | +| Praefect (Instances in ASG with load balancer) | 6 vCPU, 10 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **c5.large** x 3 nodes<br />(6 vCPU, 12 GB) | $0.09 x 3 = $0.21/hr | $0.09 x 3 = $0.21/hr | +| Praefect PostgreSQL(1) (AWS RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | N/A Reuses GitLab PostgreSQL | $0 | $0 | +| Internal Load Balancing Node | 2 vCPU, 1.8 GB | AWS ELB | $0.10/hr | $0.10/hr | ### 3K Cloud Native Hybrid on EKS @@ -176,8 +174,11 @@ If EKS node autoscaling is employed, it is likely that your average loading will **GPT Test Results** -- [5K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) -- [5K AutoScale from 25% GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) +- [3K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt) + +- [3K AutoScale from 25% GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200/3k-QuickStart-AutoScale-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_194200_results.txt) + + Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven my normal production workloads. **Deploy Now** @@ -202,7 +203,7 @@ On Demand pricing is used in this table for comparisons, but should not be used | Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/3k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/3k_users.md#cluster-topology)) = <br />4 vCPU, 15 GB | | | | **GitLab Ref Arch Raw Total K8s Node Capacity** | 32 vCPU, 56 GB | | | | One Node for Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 16 vCPU, 32GB | | | -| **Grand Total w/ Overheads Full Scale**<br />Minimum hosts = 3 | 48 vCPU, 88 GB | **c5.2xlarge** (8vcpu/16GB) x 5 nodes<br />40 vCPU, 80 GB<br />[Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt) | $1.70/hr | +| **Grand Total w/ Overheads Full Scale**<br />Minimum hosts = 3 | 48 vCPU, 88 GB | **c5.2xlarge** (8vcpu/16GB) x 5 nodes<br />40 vCPU, 80 GB<br />[Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/3K/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216/3k-QuickStart-ARM-RDS-Cache_v13-12-3-ee_2021-07-23_124216_results.txt) | $1.70/hr | | **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 24 vCPU, 48 GB | c5.2xlarge x 4 | $1.36/hr | Other combinations of node type and quantity can be used to meet the Grand Total. Due to the properties of pods, hosts that are overly small may have significant unused capacity. @@ -213,12 +214,12 @@ If EKS node autoscaling is employed, it is likely that your average loading will | Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM<br />(Directly Usable in AWS Quick Start) | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------ | | **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for perf. testing | | | -| **PostgreSQL**<br />AWS Aurora RDS Nodes Configuration (GPT tested) | 18vCPU, 36 GB <br />(across 9 nodes for PostgreSQL, PgBouncer, Consul)<br />Tested with Graviton ARM | **db.r6g.xlarge** x 3 nodes <br />(12vCPU, 96 GB) | 3 nodes x $0.52 = $1.56/hr | 3 nodes x $0.52 = $1.56/hr<br />(Aurora is always 3) | +| **PostgreSQL**<br />Amazon RDS PostgreSQL Nodes Configuration (GPT tested) | 18vCPU, 36 GB <br />(across 9 nodes for PostgreSQL, PgBouncer, Consul)<br />Tested with Graviton ARM | **db.r6g.xlarge** x 3 nodes <br />(12vCPU, 96 GB) | 3 nodes x $0.52 = $1.56/hr | 3 nodes x $0.52 = $1.56/hr | | **Redis** | 6vCPU, 18GB<br />(across 6 nodes for Redis Cache, Sentinel) | **cache.m6g.large** x 3 nodes<br />(6vCPU, 19GB) | 3 nodes x $0.15 = $0.45/hr | 2 nodes x $0.15 = $0.30/hr | -| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | +| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | | Gitaly Instances (in ASG) | 12 vCPU, 45GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **m5.large** x 3 nodes<br />(12 vCPU, 48 GB) | $0.192 x 3 = $0.58/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | | Praefect (Instances in ASG with load balancer) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **c5.large** x 3 nodes<br />(6 vCPU, 12 GB) | $0.09 x 3 = $0.21/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect PostgreSQL(1) (AWS RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | N/A Reuses GitLab PostgreSQL | $0 | | +| Praefect PostgreSQL(1) (Amazon RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | N/A Reuses GitLab PostgreSQL | $0 | | | Internal Load Balancing Node | 2 vCPU, 1.8 GB | AWS ELB | $0.10/hr | $0.10/hr | ### 5K Cloud Native Hybrid on EKS @@ -227,9 +228,12 @@ If EKS node autoscaling is employed, it is likely that your average loading will **GPT Test Results** -- [5K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) +- [5K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) + - [5K AutoScale from 25% GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717/5k-QuickStart-AutoScale-From-25Percent-ARM-RDS-Redis_v13-12-3-ee_2021-07-24_102717_results.txt) + Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven my normal production workloads. + **Deploy Now** Deploy Now links leverage the AWS Quick Start automation and only prepopulate the number of instances and instance types for the Quick Start based on the Bill of Meterials below. You must provide appropriate input for all other parameters by following the guidance in the [Quick Start documentation's Deployment steps](https://aws-quickstart.github.io/quickstart-eks-gitlab/#_deployment_steps) section. @@ -253,7 +257,7 @@ On Demand pricing is used in this table for comparisons, but should not be used | Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/5k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/5k_users.md#cluster-topology)) = <br />4 vCPU, 15 GB | | | | **GitLab Ref Arch Raw Total K8s Node Capacity** | 62 vCPU, 96.5 GB | | | | One Node for Quick Start Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 8 vCPU, 16GB | | | -| **Grand Total w/ Overheads Full Scale**<br />Minimum hosts = 3 | 70 vCPU, 112.5 GB | **c5.2xlarge** (8vcpu/16GB) x 9 nodes<br />72 vCPU, 144 GB<br />[Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | $2.38/hr | +| **Grand Total w/ Overheads Full Scale**<br />Minimum hosts = 3 | 70 vCPU, 112.5 GB | **c5.2xlarge** (8vcpu/16GB) x 9 nodes<br />72 vCPU, 144 GB<br />[Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/5K/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128/5k-QuickStart-ARM-RDS-Redis_v13-12-3-ee_2021-07-23_140128_results.txt) | $2.38/hr | | **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 24 vCPU, 48 GB | c5.2xlarge x 7 | $1.85/hr | Other combinations of node type and quantity can be used to meet the Grand Total. Due to the cpu and memory requirements of pods, hosts that are overly small may have significant unused capacity. @@ -264,12 +268,12 @@ If EKS node autoscaling is employed, it is likely that your average loading will | Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM<br />(Directly Usable in AWS Quick Start) | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | | ------------------------------------------------------------ | ------------------------------------------------------------ | ------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------ | | **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for perf. testing | | | -| **PostgreSQL**<br />AWS Aurora RDS Nodes Configuration (GPT tested) | 21vCPU, 51 GB <br />(across 9 nodes for PostgreSQL, PgBouncer, Consul)<br />Tested with Graviton ARM | **db.r6g.2xlarge** x 3 nodes <br />(24vCPU, 192 GB) | 3 nodes x $1.04 = $3.12/hr | 3 nodes x $1.04 = $3.12/hr<br />(Aurora is always 3) | +| **PostgreSQL**<br />Amazon RDS PostgreSQL Nodes Configuration (GPT tested) | 21vCPU, 51 GB <br />(across 9 nodes for PostgreSQL, PgBouncer, Consul)<br />Tested with Graviton ARM | **db.r6g.2xlarge** x 3 nodes <br />(24vCPU, 192 GB) | 3 nodes x $1.04 = $3.12/hr | 3 nodes x $1.04 = $3.12/hr | | **Redis** | 9vCPU, 27GB<br />(across 6 nodes for Redis, Sentinel) | **cache.m6g.xlarge** x 3 nodes<br />(12vCPU, 39GB) | 3 nodes x $0.30 = $0.90/hr | 2 nodes x $0.30 = $0.60/hr | -| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | +| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | | Gitaly Instances (in ASG) | 24 vCPU, 90GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **m5.2xlarge** x 3 nodes<br />(24 vCPU, 96GB) | $0.384 x 3 = $1.15/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | | Praefect (Instances in ASG with load balancer) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **c5.large** x 3 nodes<br />(6 vCPU, 12 GB) | $0.09 x 3 = $0.21/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect PostgreSQL(1) (AWS RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | N/A Reuses GitLab PostgreSQL | $0 | | +| Praefect PostgreSQL(1) (Amazon RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | N/A Reuses GitLab PostgreSQL | $0 | | | Internal Load Balancing Node | 2 vCPU, 1.8 GB | AWS ELB | $0.10/hr | $0.10/hr | ### 10K Cloud Native Hybrid on EKS @@ -278,8 +282,11 @@ If EKS node autoscaling is employed, it is likely that your average loading will **GPT Test Results** -- [10K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt) -- [10K AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) +- [10K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt) + +- [10K Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) + + Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven my normal production workloads. **Deploy Now** @@ -303,8 +310,8 @@ On Demand pricing is used in this table for comparisons, but should not be used | Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/10k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/10k_users.md#cluster-topology))<br />4 vCPU, 15 GB | | | | **GitLab Ref Arch Raw Total K8s Node Capacity** | 128 vCPU, 158 GB | | | | One Node for Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 16 vCPU, 32GB | | | -| **Grand Total w/ Overheads Fully Scaled**<br />Minimum hosts = 3 | 142 vCPU, 190 GB | **c5.4xlarge** (16vcpu/32GB) x 9 nodes<br />144 vCPU, 288GB<br /><br />[Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt) | $6.12/hr | -| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 40 vCPU, 80 GB | c5.4xlarge x 7<br /><br />[AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | $4.76/hr | +| **Grand Total w/ Overheads Fully Scaled**<br />Minimum hosts = 3 | 142 vCPU, 190 GB | **c5.4xlarge** (16vcpu/32GB) x 9 nodes<br />144 vCPU, 288GB<br /><br />[Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647/GL-CloudNative-10k-RDS-Graviton_v13-12-3-ee_2021-07-08_194647_results.txt) | $6.12/hr | +| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 40 vCPU, 80 GB | c5.4xlarge x 7<br /><br />[Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/10K/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139/GL-CloudNative-10k-AutoScaling-Test_v13-12-3-ee_2021-07-09_115139_results.txt) | $4.76/hr | Other combinations of node type and quantity can be used to meet the Grand Total. Due to the cpu and memory requirements of pods, hosts that are overly small may have significant unused capacity. @@ -314,12 +321,12 @@ If EKS node autoscaling is employed, it is likely that your average loading will | Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | | ------------------------------------------------------------ | ------------------------------ | ------------------------------- | ------------------------------------------------------------ | ------------------------------------------------------------ | | **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for perf. testing | | | -| **PostgreSQL**<br />AWS Aurora RDS Nodes Configuration (GPT tested) | 36vCPU, 102 GB <br />(across 9 nodes for PostgreSQL, PgBouncer, Consul) | **db.r6g.2xlarge** x 3 nodes <br />(24vCPU, 192 GB) | 3 nodes x $1.04 = $3.12/hr | 3 nodes x $1.04 = $3.12/hr<br />(Aurora is always 3) | +| **PostgreSQL**<br />Amazon RDS PostgreSQL Nodes Configuration (GPT tested) | 36vCPU, 102 GB <br />(across 9 nodes for PostgreSQL, PgBouncer, Consul) | **db.r6g.2xlarge** x 3 nodes <br />(24vCPU, 192 GB) | 3 nodes x $1.04 = $3.12/hr | 3 nodes x $1.04 = $3.12/hr | | **Redis** | 30vCPU, 114GB<br />(across 12 nodes for Redis Cache, Redis Queues/Shared State, Sentinel Cache, Sentinel Queues/Shared State) | **cache.m5.2xlarge** x 3 nodes<br />(24vCPU, 78GB) | 3 nodes x $0.62 = $1.86/hr | 2 nodes x $0.62 = $1.24/hr | | **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | | Gitaly Instances (in ASG) | 48 vCPU, 180GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **m5.4xlarge** x 3 nodes<br />(48 vCPU, 180 GB) | $0.77 x 3 = $2.31/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | | Praefect (Instances in ASG with load balancer) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | **c5.large** x 3 nodes<br />(6 vCPU, 12 GB) | $0.09 x 3 = $0.21/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | -| Praefect PostgreSQL(1) (AWS RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | N/A Reuses GitLab PostgreSQL | $0 | | +| Praefect PostgreSQL(1) (Amazon RDS) | 6 vCPU, 5.4 GB<br />([across 3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections)) | N/A Reuses GitLab PostgreSQL | $0 | | | Internal Load Balancing Node | 2 vCPU, 1.8 GB | AWS ELB | $0.10/hr | $0.10/hr | ### 50K Cloud Native Hybrid on EKS @@ -328,8 +335,11 @@ If EKS node autoscaling is employed, it is likely that your average loading will **GPT Test Results** -- [50K Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt) -- [50K AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) +- [50K Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt) + +- [50K Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) + + Elastic Auto Scale GPT Test Results start with an idle scaled cluster and then start the standard GPT test to determine if the EKS Auto Scaler performs well enough to keep up with performance test demands. In general this is substantially harder ramping than the scaling required when the ramping is driven my normal production workloads. **Deploy Now** @@ -353,8 +363,8 @@ On Demand pricing is used in this table for comparisons, but should not be used | Supporting services such as NGINX, Prometheus, etc | [2 allocations](../../administration/reference_architectures/10k_users.md#cluster-topology) x ([2 vCPU and 7.5 GB](../../administration/reference_architectures/10k_users.md#cluster-topology))<br />4 vCPU, 15 GB | | | | **GitLab Ref Arch Raw Total K8s Node Capacity** | 428 vCPU, 533 GB | | | | One Node for Overhead and Miscellaneous (EKS Cluster AutoScaler, Grafana, Prometheus, etc) | + 16 vCPU, 32GB | | | -| **Grand Total w/ Overheads Fully Scaled**<br />Minimum hosts = 3 | 444 vCPU, 565 GB | **c5.4xlarge** (16vcpu/32GB) x 28 nodes<br />448 vCPU, 896GB<br /><br />[Full Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt) | $19.04/hr | -| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 40 vCPU, 80 GB | c5.2xlarge x 10<br /><br />[AutoScale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | $6.80/hr | +| **Grand Total w/ Overheads Fully Scaled**<br />Minimum hosts = 3 | 444 vCPU, 565 GB | **c5.4xlarge** (16vcpu/32GB) x 28 nodes<br />448 vCPU, 896GB<br /><br />[Full Fixed Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819/50k-Fixed-Scale-Test_v13-12-3-ee_2021-08-13_172819_results.txt) | $19.04/hr | +| **Possible Idle Configuration (Scaled-In 75% - round up)**<br />Pod autoscaling must be also adjusted to enable lower idling configuration. | 40 vCPU, 80 GB | c5.2xlarge x 10<br /><br />[Elastic Auto Scale GPT Test Results](https://gitlab.com/gitlab-com/alliances/aws/implementation-patterns/gitlab-cloud-native-hybrid-on-eks/-/blob/master/gitlab-alliances-testing/50K/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633/50k-AutoScale-Test_v13-12-3-ee_2021-08-13_192633.txt) | $6.80/hr | Other combinations of node type and quantity can be used to meet the Grand Total. Due to the cpu and memory requirements of pods, hosts that are overly small may have significant unused capacity. @@ -364,9 +374,9 @@ If EKS node autoscaling is employed, it is likely that your average loading will | Non-Kubernetes Compute | Ref Arch Raw Total | AWS BOM | Example Cost<br />US East, 3 AZ | Example Cost<br />US East, 2 AZ | | ------------------------------------------------------------ | ------------------------------------------------------------ | --------------------------------------------------------- | ------------------------------- | ------------------------------------------------------------ | | **Bastion Host (Quick Start)** | 1 HA instance in ASG | **t2.micro** for prod, **m4.2xlarge** for perf. testing | | | -| **PostgreSQL**<br />AWS Aurora RDS Nodes Configuration (GPT tested) | 96vCPU, 360 GB <br />(across 3 nodes) | **db.r6g.8xlarge** x 3 nodes <br />(96vCPU, 768 GB total) | 3 nodes x $4.15 = $12.45/hr | 3 nodes x $4.15 = $12.45/hr<br />(Aurora is always 3) | +| **PostgreSQL**<br />Amazon RDS PostgreSQL Nodes Configuration (GPT tested) | 96vCPU, 360 GB <br />(across 3 nodes) | **db.r6g.8xlarge** x 3 nodes <br />(96vCPU, 768 GB total) | 3 nodes x $4.15 = $12.45/hr | 3 nodes x $4.15 = $12.45/hr | | **Redis** | 30vCPU, 114GB<br />(across 12 nodes for Redis Cache, Redis Queues/Shared State, Sentinel Cache, Sentinel Queues/Shared State) | **cache.m6g.2xlarge** x 3 nodes<br />(24vCPU, 78GB total) | 3 nodes x $0.60 = $1.80/hr | 2 nodes x $0.60 = $1.20/hr | -| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | +| **<u>Gitaly Cluster</u>** [Details](gitlab_sre_for_aws.md#gitaly-sre-considerations) | | | | | | Gitaly Instances (in ASG) | 64 vCPU, 240GB x [3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | **m5.16xlarge** x 3 nodes<br />(64 vCPU, 256 GB each) | $3.07 x 3 = $9.21/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | | Praefect (Instances in ASG with load balancer) | 4 vCPU, 3.6 GB x [3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | **c5.xlarge** x 3 nodes<br />(4 vCPU, 8 GB each) | $0.17 x 3 = $0.51/hr | [Gitaly & Praefect Must Have an Uneven Node Count for HA](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | | Praefect PostgreSQL(1) (AWS RDS) | 2 vCPU, 1.8 GB x [3 nodes](gitlab_sre_for_aws.md#gitaly-and-praefect-elections) | N/A Reuses GitLab PostgreSQL | $0 | | @@ -375,3 +385,11 @@ If EKS node autoscaling is employed, it is likely that your average loading will ## Helpful Resources - [Architecting Kubernetes clusters — choosing a worker node size](https://learnk8s.io/kubernetes-node-size) + +DISCLAIMER: +This page contains information related to upcoming products, features, and functionality. +It is important to note that the information presented is for informational purposes only. +Please do not rely on this information for purchasing or planning purposes. +As with all projects, the items mentioned on this page are subject to change or delay. +The development, release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. diff --git a/doc/install/aws/gitlab_sre_for_aws.md b/doc/install/aws/gitlab_sre_for_aws.md index 06e3bf784bd..faec73801ab 100644 --- a/doc/install/aws/gitlab_sre_for_aws.md +++ b/doc/install/aws/gitlab_sre_for_aws.md @@ -1,20 +1,13 @@ --- stage: Enablement -group: Alliances +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false description: Doing SRE for GitLab instances and runners on AWS. -type: index --- # GitLab Site Reliability Engineering for AWS **(FREE SELF)** -## AWS known issues list - -Known issues are gathered from within GitLab and from customer reported issues. Customers successfully implement GitLab with a variety of "as a Service" components that GitLab has not specifically been designed for, nor has ongoing testing for. While GitLab does take partner technologies very seriously, the highlighting of known issues here is a convenience for implementers and it does not imply that GitLab has targeted compatibility with, nor carries any type of guarantee of running on the partner technology where the issues occur. Please consult individual issues to understand GitLabs stance and plans on any given known issue. - -See the [GitLab AWS known issues list](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues?label_name%5B%5D=AWS+Known+Issue) for a complete list. - ## Gitaly SRE considerations Gitaly is an embedded service for Git Repository Storage. Gitaly and Gitaly Cluster have been engineered by GitLab to overcome fundamental challenges with horizontal scaling of the open source Git binaries that must be used on the service side of GitLab. Here is indepth technical reading on the topic: diff --git a/doc/install/aws/index.md b/doc/install/aws/index.md index 80193d882dd..563673c3260 100644 --- a/doc/install/aws/index.md +++ b/doc/install/aws/index.md @@ -1,6 +1,6 @@ --- stage: Enablement -group: Alliances +group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments comments: false description: Read through the GitLab installation methods. @@ -11,30 +11,38 @@ type: index GitLab [Reference Architectures](../../administration/reference_architectures/index.md) give qualified and tested guidance on the recommended ways GitLab can be configured to meet the performance requirements of various workloads. Reference Architectures are purpose-designed to be non-implementation specific so they can be extrapolated to as many environments as possible. This generally means they have a highly-granular "machine" to "server role" specification and focus on system elements that impact performance. This is what enables Reference Architectures to be adaptable to the broadest number of supported implementations. -Implementation patterns are built on the foundational information and testing done for Reference Architectures and allow architects and implementers at GitLab, GitLab Customers, and GitLab Partners to build out deployments with less experimentation and a higher degree of confidence that the results will perform as expected. +Implementation patterns are built on the foundational information and testing done for Reference Architectures and allow architects and implementers at GitLab, GitLab Customers, and GitLab Partners to build out deployments with less experimentation and a higher degree of confidence that the results will perform as expected. A more thorough discussion of implementation patterns is below in [Additional details on implementation patterns](#additional-details-on-implementation-patterns). -## Implementation patterns information +## AWS Implementation patterns information -### Install GitLab Cloud Native Hybrid on AWS EKS (HA) +The following are the currently available implementation patterns for GitLab when it is implemented on AWS. -[Provision GitLab Cloud Native Hybrid on AWS EKS (HA)](gitlab_hybrid_on_aws.md). This document includes instructions, patterns, and automation for installing GitLab Cloud Native Hybrid on AWS EKS. It also includes [Bill of Materials](https://en.wikipedia.org/wiki/Bill_of_materials) listings and links to Infrastructure as Code. GitLab Cloud Native Hybrid is the supported way to put as much of GitLab as possible into Kubernetes. +### GitLab Site Reliability Engineering (SRE) for AWS -### Install Omnibus GitLab on AWS EC2 (HA) +[GitLab Site Reliability Engineering (SRE) for AWS](gitlab_sre_for_aws.md) - information for planning, implementing, upgrading and long term management of GitLab instances and runners on AWS. -[Omnibus GitLab on AWS EC2 (HA)](manual_install_aws.md) - instructions for installing GitLab on EC2 instances. Manual instructions from which you may build out a GitLab instance or create your own Infrastructure as Code (IaC). +### Patterns to Install GitLab Cloud Native Hybrid on AWS EKS (HA) -### GitLab Site Reliability Engineering (SRE) for AWS +[Provision GitLab Cloud Native Hybrid on AWS EKS (HA)](gitlab_hybrid_on_aws.md). This document includes instructions, patterns, and automation for installing GitLab Cloud Native Hybrid on AWS EKS. It also includes [Bill of Materials](https://en.wikipedia.org/wiki/Bill_of_materials) listings and links to Infrastructure as Code. GitLab Cloud Native Hybrid is the supported way to put as much of GitLab as possible into Kubernetes. -[GitLab SRE Considerations for AWS](gitlab_sre_for_aws.md) - important information and known issues for planning, implementing, upgrading and long term management of GitLab instances and runners on AWS. +### Patterns to Install Omnibus GitLab on AWS EC2 (HA) -### EKS cluster provisioning best practices +[Omnibus GitLab on AWS EC2 (HA)](manual_install_aws.md) - instructions for installing GitLab on EC2 instances. Manual instructions to build a GitLab instance or create your own Infrastructure as Code (IaC). + +### Patterns for EKS cluster provisioning [EKS Cluster Provisioning Patterns](eks_clusters_aws.md) - considerations for setting up EKS cluster for runners and for integrating. -### Scaling HA GitLab Runner on AWS EC2 ASG +### Patterns for Scaling HA GitLab Runner on AWS EC2 ASG The following repository is self-contained in regard to enabling this pattern: [GitLab HA Scaling Runner Vending Machine for AWS EC2 ASG](https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/). The [feature list for this implementation pattern](https://gitlab.com/guided-explorations/aws/gitlab-runner-autoscaling-aws-asg/-/blob/main/FEATURES.md) is good to review to understand the complete value it can deliver. +## AWS known issues list + +Known issues are gathered from within GitLab and from customer reported issues. Customers successfully implement GitLab with a variety of "as a Service" components that GitLab has not specifically been designed for, nor has ongoing testing for. While GitLab does take partner technologies very seriously, the highlighting of known issues here is a convenience for implementers and it does not imply that GitLab has targeted compatibility with, nor carries any type of guarantee of running on the partner technology where the issues occur. Please consult individual issues to understand GitLabs stance and plans on any given known issue. + +See the [GitLab AWS known issues list](https://gitlab.com/gitlab-com/alliances/aws/public-tracker/-/issues?label_name%5B%5D=AWS+Known+Issue) for a complete list. + ## Additional details on implementation patterns GitLab implementation patterns build upon [GitLab Reference Architectures](../../administration/reference_architectures/index.md) in the following ways. @@ -61,7 +69,7 @@ Implementation patterns enable platform-specific terminology, best practice arch Platform as a Service options are a huge portion of the value provided by Cloud Platforms as they simplify operational complexity and reduce the SRE and security skilling required to operate advanced, highly available technology services. Implementation patterns can be prequalified against the partner PaaS options. -- Implementation patterns help implementers understand what PaaS options are known to work and how to choose between PaaS solutions when a single platform has more than one PaaS option for the same GitLab role (for example, AWS RDS versus AWS Aurora RDS). +- Implementation patterns help implementers understand what PaaS options are known to work and how to choose between PaaS solutions when a single platform has more than one PaaS option for the same GitLab role. - For instance, where reference architectures do not have a specific recommendation on what technology is leveraged for GitLab outbound email services or what the sizing should be - a Reference Implementation may advise using a cloud providers Email as a Service (PaaS) and possibly even with specific settings. ### Cost optimizing engineering diff --git a/doc/install/aws/manual_install_aws.md b/doc/install/aws/manual_install_aws.md index 2ad54a17715..6fff4225346 100644 --- a/doc/install/aws/manual_install_aws.md +++ b/doc/install/aws/manual_install_aws.md @@ -2,7 +2,6 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- {::options parse_block_html="true" /} @@ -33,7 +32,7 @@ For the Cloud Native Hybrid architectures there are two Infrastructure as Code o ## Introduction -For the most part, we'll make use of Omnibus GitLab in our setup, but we'll also leverage native AWS services. Instead of using the Omnibus bundled PostgreSQL and Redis, we will use AWS RDS and ElastiCache. +For the most part, we'll make use of Omnibus GitLab in our setup, but we'll also leverage native AWS services. Instead of using the Omnibus bundled PostgreSQL and Redis, we will use Amazon RDS and ElastiCache. In this guide, we'll go through a multi-node setup where we'll start by configuring our Virtual Private Cloud and subnets to later integrate @@ -410,7 +409,10 @@ persistence and is used to store session data, temporary cache information, and ## Setting up Bastion Hosts -Since our GitLab instances will be in private subnets, we need a way to connect to these instances via SSH to make configuration changes, perform upgrades, etc. One way of doing this is via a [bastion host](https://en.wikipedia.org/wiki/Bastion_host), sometimes also referred to as a jump box. +Because our GitLab instances are in private subnets, we need a way to connect +to these instances with SSH for actions that include making configuration changes +and performing upgrades. One way of doing this is by using a [bastion host](https://en.wikipedia.org/wiki/Bastion_host), +sometimes also referred to as a jump box. NOTE: If you do not want to maintain bastion hosts, you can set up [AWS Systems Manager Session Manager](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager.html) for access to instances. This is beyond the scope of this document. @@ -669,8 +671,13 @@ That concludes the configuration changes for our GitLab instance. Next, we'll cr ### Log in for the first time Using the domain name you used when setting up [DNS for the load balancer](#configure-dns-for-load-balancer), you should now be able to visit GitLab in your browser. -If you didn't change the password by any other means, the default password will be the same as the instance ID. To change the default password, login as the `root` user -with the default password and [change it in the user profile](../../user/profile#change-your-password). + +Depending on how you installed GitLab and if you did not change the password by any other means, the default password is either: + +- Your instance ID if you used the official GitLab AMI. +- A randomly generated password stored for 24 hours in `/etc/gitlab/initial_root_password`. + +To change the default password, log in as the `root` user with the default password and [change it in the user profile](../../user/profile#change-your-password). When our [auto scaling group](#create-an-auto-scaling-group) spins up new instances, we'll be able to log in with username `root` and the newly created password. diff --git a/doc/install/docker.md b/doc/install/docker.md index b3e7e758ec3..00e19e2977b 100644 --- a/doc/install/docker.md +++ b/doc/install/docker.md @@ -27,7 +27,7 @@ WARNING: Docker for Windows is not officially supported. There are known issues with volume permissions, and potentially other unknown issues. If you are trying to run on Docker for Windows, see the [getting help page](https://about.gitlab.com/get-help/) for links -to community resources (IRC, forum, etc.) to seek help from other users. +to community resources (such as IRC or forums) to seek help from other users. ## Prerequisites diff --git a/doc/install/google_cloud_platform/index.md b/doc/install/google_cloud_platform/index.md index bda98ead1f5..b3d0863f6a3 100644 --- a/doc/install/google_cloud_platform/index.md +++ b/doc/install/google_cloud_platform/index.md @@ -3,7 +3,6 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments description: 'Learn how to install a GitLab instance on Google Cloud Platform.' -type: howto --- # Installing GitLab on Google Cloud Platform **(FREE SELF)** @@ -11,9 +10,11 @@ type: howto This guide will help you install GitLab on a [Google Cloud Platform (GCP)](https://cloud.google.com/) using the official GitLab Linux package. You should customize it to accommodate your needs. NOTE: -Google provides a whitepaper for [deploying production-ready GitLab on -Google Kubernetes Engine](https://cloud.google.com/architecture/deploying-production-ready-gitlab-on-gke), -including all steps and external resource configuration. These are an alternative to using a GCP VM, and use +To deploy production-ready GitLab on +Google Kubernetes Engine, +you can follow Google Cloud Platform's +[Click to Deploy steps](https://github.com/GoogleCloudPlatform/click-to-deploy/blob/master/k8s/gitlab/README.md) +It's an alternative to using a GCP VM, and uses the [Cloud native GitLab Helm chart](https://docs.gitlab.com/charts/). ## Prerequisites @@ -126,8 +127,8 @@ Check the [Omnibus documentation](https://docs.gitlab.com/omnibus/settings/smtp. ## Further reading -GitLab can be configured to authenticate with other OAuth providers, LDAP, SAML, -Kerberos, etc. Here are some documents you might be interested in reading: +GitLab can be configured to authenticate with other OAuth providers, like LDAP, +SAML, and Kerberos. Here are some documents you might be interested in reading: - [Omnibus GitLab documentation](https://docs.gitlab.com/omnibus/) - [Integration documentation](../../integration/index.md) diff --git a/doc/install/installation.md b/doc/install/installation.md index 7c8718a7005..cd00593a99b 100644 --- a/doc/install/installation.md +++ b/doc/install/installation.md @@ -2,7 +2,6 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: howto --- # Installation from source **(FREE SELF)** @@ -424,6 +423,49 @@ echo 'unixsocket /var/run/redis/redis.sock' | sudo tee -a /etc/redis/redis.conf # Grant permission to the socket to all members of the redis group echo 'unixsocketperm 770' | sudo tee -a /etc/redis/redis.conf +# Add git to the redis group +sudo usermod -aG redis git +``` + +### Supervise Redis with systemd + +If your distribution uses systemd init and the output of the following command is `notify`, +you do not need to make any changes: + +```shell +systemctl show --value --property=Type redis-server.service +``` + +If the output is **not** `notify`, run: + +```shell +# Configure Redis to not daemonize, but be supervised by systemd instead and disable the pidfile +sudo sed -i \ + -e 's/^daemonize yes$/daemonize no/' \ + -e 's/^supervised no$/supervised systemd/' \ + -e 's/^pidfile/# pidfile/' /etc/redis/redis.conf +sudo chown redis:redis /etc/redis/redis.conf + +# Make the same changes to the systemd unit file +sudo mkdir -p /etc/systemd/system/redis-server.service.d +sudo tee /etc/systemd/system/redis-server.service.d/10fix_type.conf <<EOF +[Service] +Type=notify +PIDFile= +EOF + +# Reload the redis service +sudo systemctl daemon-reload + +# Activate the changes to redis.conf +sudo systemctl restart redis-server.service +``` + +### Leave Redis unsupervised + +If your system uses SysV init, run these commands: + +```shell # Create the directory which contains the socket sudo mkdir -p /var/run/redis sudo chown redis:redis /var/run/redis @@ -436,9 +478,6 @@ fi # Activate the changes to redis.conf sudo service redis-server restart - -# Add git to the redis group -sudo usermod -aG redis git ``` ## 8. GitLab @@ -688,48 +727,79 @@ sudo -u git -H editor config.toml For more information about configuring Gitaly see [the Gitaly documentation](../administration/gitaly/index.md). -### Start Gitaly +### Install the service -Gitaly must be running for the next section. +GitLab has always supported SysV init scripts, which are widely supported and portable, but now systemd is the standard for service supervision and is used by all major Linux distributions. You should use native systemd services if you can to benefit from automatic restarts, better sandboxing and resource control. -```shell -gitlab_path=/home/git/gitlab -gitaly_path=/home/git/gitaly +#### Install systemd units -sudo -u git -H sh -c "$gitlab_path/bin/daemon_with_pidfile $gitlab_path/tmp/pids/gitaly.pid \ - $gitaly_path/_build/bin/gitaly $gitaly_path/config.toml >> $gitlab_path/log/gitaly.log 2>&1 &" -``` +Use these steps if you use systemd as init. Otherwise, follow the [SysV init script steps](#install-sysv-init-script). -### Initialize Database and Activate Advanced Features +Copy the services and run `systemctl daemon-reload` so that systemd picks them up: ```shell cd /home/git/gitlab -sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production -# Type 'yes' to create the database tables. +sudo mkdir -p /usr/local/lib/systemd/system +sudo cp lib/support/systemd/* /usr/local/lib/systemd/system/ +sudo systemctl daemon-reload +``` -# or you can skip the question by adding force=yes -sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production force=yes +The units provided by GitLab make very little assumptions about where you are running Redis and PostgreSQL. -# When done, you see 'Administrator account created:' -``` +If you installed GitLab in another directory or as a user other than the default, you need to change these values in the units as well. -You can set the Administrator/root password and email by supplying them in environmental variables, `GITLAB_ROOT_PASSWORD` and `GITLAB_ROOT_EMAIL` respectively, as seen below. If you don't set the password (and it is set to the default one), wait to expose GitLab to the public internet until the installation is done and you've logged into the server the first time. During the first login, you'll be forced to change the default password. An Enterprise Edition license may also be installed at this time by supplying a full path in the `GITLAB_LICENSE_FILE` environment variable. +For example, if you're running Redis and PostgreSQL on the same machine as GitLab, you should: + +- Edit the Puma service: + + ```shell + sudo systemctl edit gitlab-puma.service + ``` + + In the editor that opens, add the following and save the file: + + ```plaintext + [Unit] + Wants=redis-server.service postgresql.service + After=redis-server.service postgresql.service + ``` + +- Edit the Sidekiq service: + + ```shell + sudo systemctl edit gitlab-sidekiq.service + ``` + + Add the following and save the file: + + ```plaintext + [Unit] + Wants=redis-server.service postgresql.service + After=redis-server.service postgresql.service + ``` + +`systemctl edit` installs drop-in configuration files at `/etc/systemd/system/<name of the unit>.d/override.conf`, so your local configuration is not overwritten when updating the unit files later. To split up your drop-in configuration files, you can add the above snippets to `.conf` files under `/etc/systemd/system/<name of the unit>.d/`. + +If you manually made changes to the unit files or added drop-in configuration files (without using `systemctl edit`), run the following command for them to take effect: ```shell -sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=yourpassword GITLAB_ROOT_EMAIL=youremail GITLAB_LICENSE_FILE="/path/to/license" +sudo systemctl daemon-reload ``` -### Secure secrets.yml +Make GitLab start on boot: -The `secrets.yml` file stores encryption keys for sessions and secure variables. -Backup `secrets.yml` someplace safe, but don't store it in the same place as your database backups. -Otherwise, your secrets are exposed if one of your backups is compromised. +```shell +sudo systemctl enable gitlab.target +``` + +#### Install SysV init script -### Install Init Script +Use these steps if you use the SysV init script. If you use systemd, follow the [systemd unit steps](#install-systemd-units). Download the init script (is `/etc/init.d/gitlab`): ```shell +cd /home/git/gitlab sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab ``` @@ -745,6 +815,9 @@ Make GitLab start on boot: ```shell sudo update-rc.d gitlab defaults 21 +# or if running this on a machine running systemd +sudo systemctl daemon-reload +sudo systemctl enable gitlab.service ``` ### Set up Logrotate @@ -753,6 +826,51 @@ sudo update-rc.d gitlab defaults 21 sudo cp lib/support/logrotate/gitlab /etc/logrotate.d/gitlab ``` +### Start Gitaly + +Gitaly must be running for the next section. + +- To start Gitaly using systemd: + + ```shell + sudo systemctl start gitlab-gitaly.service + ``` + +- To manually start Gitaly for SysV: + + ```shell + gitlab_path=/home/git/gitlab + gitaly_path=/home/git/gitaly + + sudo -u git -H sh -c "$gitlab_path/bin/daemon_with_pidfile $gitlab_path/tmp/pids/gitaly.pid \ + $gitaly_path/_build/bin/gitaly $gitaly_path/config.toml >> $gitlab_path/log/gitaly.log 2>&1 &" + ``` + +### Initialize Database and Activate Advanced Features + +```shell +cd /home/git/gitlab +sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production +# Type 'yes' to create the database tables. + +# or you can skip the question by adding force=yes +sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production force=yes + +# When done, you see 'Administrator account created:' +``` + +You can set the Administrator/root password and email by supplying them in environmental variables, `GITLAB_ROOT_PASSWORD` and `GITLAB_ROOT_EMAIL` respectively, as seen below. If you don't set the password (and it is set to the default one), wait to expose GitLab to the public internet until the installation is done and you've logged into the server the first time. During the first login, you'll be forced to change the default password. An Enterprise Edition license may also be installed at this time by supplying a full path in the `GITLAB_LICENSE_FILE` environment variable. + +```shell +sudo -u git -H bundle exec rake gitlab:setup RAILS_ENV=production GITLAB_ROOT_PASSWORD=yourpassword GITLAB_ROOT_EMAIL=youremail GITLAB_LICENSE_FILE="/path/to/license" +``` + +### Secure secrets.yml + +The `secrets.yml` file stores encryption keys for sessions and secure variables. +Backup `secrets.yml` someplace safe, but don't store it in the same place as your database backups. +Otherwise, your secrets are exposed if one of your backups is compromised. + ### Check Application Status Check if GitLab and its environment are configured correctly: @@ -783,9 +901,11 @@ sudo -u git -H bundle exec rake gitlab:assets:compile RAILS_ENV=production NODE_ ### Start Your GitLab Instance ```shell +# For systems running systemd +sudo systemctl start gitlab.target + +# For systems running SysV init sudo service gitlab start -# or -sudo /etc/init.d/gitlab restart ``` ## 9. NGINX @@ -838,7 +958,9 @@ Validate your `gitlab` or `gitlab-ssl` NGINX configuration file with the followi sudo nginx -t ``` -You should receive `syntax is okay` and `test is successful` messages. If you receive errors check your `gitlab` or `gitlab-ssl` NGINX configuration file for typos, etc. as indicated in the error message given. +You should receive `syntax is okay` and `test is successful` messages. If you +receive error messages, check your `gitlab` or `gitlab-ssl` NGINX configuration +file for typos, as indicated in the provided error message. Verify that the installed version is greater than 1.12.1: @@ -856,6 +978,10 @@ nginx: configuration file /etc/nginx/nginx.conf test failed ### Restart ```shell +# For systems running systemd +sudo systemctl restart nginx.service + +# For systems running SysV init sudo service nginx restart ``` @@ -888,7 +1014,10 @@ earlier and login. After login, you can change the username if you wish. **Enjoy!** -You can use `sudo service gitlab start` and `sudo service gitlab stop` to start and stop GitLab. +To start and stop GitLab when using: + +- systemd units: use `sudo systemctl start gitlab.target` or `sudo systemctl stop gitlab.target`. +- The SysV init script: use `sudo service gitlab start` or `sudo service gitlab stop`. ## Advanced Setup Tips @@ -1026,7 +1155,7 @@ misconfigured GitLab Workhorse instance. Double-check that you've [installed Go](#3-go), [installed GitLab Workhorse](#install-gitlab-workhorse), and correctly [configured NGINX](#site-configuration). -### `google-protobuf` "LoadError: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.14' not found" +### `google-protobuf` "LoadError: /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.14' not found" This can happen on some platforms for some versions of the `google-protobuf` gem. The workaround is to install a source-only diff --git a/doc/install/openshift_and_gitlab/index.md b/doc/install/openshift_and_gitlab/index.md index 3b7ea5c1975..f74e9f26362 100644 --- a/doc/install/openshift_and_gitlab/index.md +++ b/doc/install/openshift_and_gitlab/index.md @@ -1,9 +1,45 @@ --- -redirect_to: 'https://docs.gitlab.com/charts/installation/operator.html' -remove_date: '2022-09-22' +stage: Enablement +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -This file was moved to [another location](https://docs.gitlab.com/charts/installation/operator.html). +# OpenShift support -<!-- This redirect file can be deleted after <2022-09-22>. --> -<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> +OpenShift - GitLab compatibility can be addressed in three different aspects. This page helps navigating between these aspects and provides introductory information for getting started with OpenShift and GitLab. + +## What is OpenShift + +OpenShift helps you to develop, deploy, and manage container-based applications. It provides you with a self-service platform to create, modify, and deploy applications on demand, thus enabling faster development and release life cycles. + +## Use OpenShift to run GitLab Self-Managed + +Running GitLab within an OpenShift cluster is officially supported using the GitLab Operator. You can learn more on +[setting up GitLab on OpenShift on the GitLab Operator's documentation](https://docs.gitlab.com/charts/installation/operator.html). +Some components (documented on the GitLab Operator doc) are not supported yet. + +## Deploy to and integrate with OpenShift from GitLab + +Deploying custom or COTS applications on top of OpenShift from GitLab is supported using [the GitLab Kubernetes Agent](../../user/clusters/agent/index.md). + +## Use OpenShift to run a GitLab Runner Fleet + +The GitLab Operator does not include the GitLab Runner. To install and manage a GitLab Runner fleet in an OpenShift cluster, use the +[GitLab Runner Operator](https://gitlab.com/gitlab-org/gl-openshift/gitlab-runner-operator). + +## Unsupported GitLab features + +### Docker-in-Docker + +When using OpenShift to run a GitLab Runner Fleet, we do not support some GitLab features given OpenShift's security model. +Features requiring Docker-in-Docker might not work. + +For Auto DevOps, the following features are not supported yet: + +- Auto Code Quality +- Auto License Compliance +- Auto Browser Performance Testing +- Auto Build + +For Auto Build, there's a [possible workaround using `kaniko`](../../ci/docker/using_kaniko.md). +You can check the progress of the implementation in this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/332560). diff --git a/doc/install/relative_url.md b/doc/install/relative_url.md index 8b629e9084e..569f6e02ea4 100644 --- a/doc/install/relative_url.md +++ b/doc/install/relative_url.md @@ -2,7 +2,6 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Install GitLab under a relative URL **(FREE SELF)** @@ -37,11 +36,11 @@ After all the changes you need to recompile the assets and [restart GitLab](../a ## Relative URL requirements -If you configure GitLab with a relative URL, the assets (JavaScript, CSS, fonts, -images, etc.) will need to be recompiled, which is a task which consumes a lot -of CPU and memory resources. To avoid out-of-memory errors, you should have at -least 2GB of RAM available on your system, while we recommend 4GB RAM, and 4 or -8 CPU cores. +If you configure GitLab with a relative URL, the assets (including JavaScript, +CSS, fonts, and images) must be recompiled, which can consume a lot of CPU and +memory resources. To avoid out-of-memory errors, you should have at least 2 GB +of RAM available on your computer, and we recommend 4 GB RAM, and four or eight +CPU cores. See the [requirements](requirements.md) document for more information. @@ -102,8 +101,9 @@ Make sure to follow all steps below: gitlab_url: http://127.0.0.1/gitlab ``` -1. Make sure you have copied the supplied init script and the defaults file - as stated in the [installation guide](installation.md#install-init-script). +1. Make sure you have copied either the supplied systemd services, or the init + script and the defaults file, as stated in the + [installation guide](installation.md#install-the-service). Then, edit `/etc/default/gitlab` and set in `gitlab_workhorse_options` the `-authBackend` setting to read like: diff --git a/doc/install/requirements.md b/doc/install/requirements.md index c136fb21a90..5dbe9a1154f 100644 --- a/doc/install/requirements.md +++ b/doc/install/requirements.md @@ -2,7 +2,6 @@ stage: Enablement group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference --- # Installation requirements **(FREE SELF)** @@ -119,7 +118,6 @@ the following table) as these were used for development and testing: | GitLab version | Minimum PostgreSQL version | |----------------|----------------------------| -| 10.0 | 9.6 | | 13.0 | 11 | | 14.0 | 12 | @@ -272,9 +270,9 @@ On a very active server (10,000 billable users) the Sidekiq process can use 1GB+ ## Prometheus and its exporters -As of Omnibus GitLab 9.0, [Prometheus](https://prometheus.io) and its related -exporters are enabled by default, to enable easy and in depth monitoring of -GitLab. With default settings, these processes consume approximately 200MB of memory. +[Prometheus](https://prometheus.io) and its related exporters are enabled by +default to enable in depth monitoring of GitLab. With default settings, these +processes consume approximately 200 MB of memory. If you would like to disable Prometheus and it's exporters or read more information about it, check the [Prometheus documentation](../administration/monitoring/prometheus/index.md). @@ -304,7 +302,7 @@ The GitLab Runner server requirements depend on: Since the nature of the jobs varies for each use case, you need to experiment by adjusting the job concurrency to get the optimum setting. -For reference, the GitLab.com Build Cloud [auto-scaling runner for Linux](../ci/runners/build_cloud/linux_build_cloud.md) is configured so that a **single job** runs in a **single instance** with: +For reference, the GitLab.com Runner Cloud [auto-scaling runner for Linux](../ci/runners/build_cloud/linux_build_cloud.md) is configured so that a **single job** runs in a **single instance** with: - 1 vCPU. - 3.75 GB of RAM. diff --git a/doc/integration/akismet.md b/doc/integration/akismet.md index d216b827676..d5e39a59dff 100644 --- a/doc/integration/akismet.md +++ b/doc/integration/akismet.md @@ -8,7 +8,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w GitLab uses [Akismet](https://akismet.com/) to prevent the creation of spam issues on public projects. Issues created through the web UI or the API can be submitted to -Akismet for review. +Akismet for review, and instance administrators can +[mark snippets as spam](../user/snippets.md#mark-snippet-as-spam). Detected spam is rejected, and an entry is added in the **Spam Log** section of the Admin page. diff --git a/doc/integration/auth0.md b/doc/integration/auth0.md index 870da6cdb3d..e243e1defe8 100644 --- a/doc/integration/auth0.md +++ b/doc/integration/auth0.md @@ -48,7 +48,7 @@ application. sudo -u git -H editor config/gitlab.yml ``` -1. Read [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) +1. Read [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration: diff --git a/doc/integration/azure.md b/doc/integration/azure.md index 3eb0344edda..bd16c8c0069 100644 --- a/doc/integration/azure.md +++ b/doc/integration/azure.md @@ -48,7 +48,7 @@ As you go through the Microsoft procedure, keep the following in mind: sudo -u git -H editor config/gitlab.yml ``` -1. Refer to [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) +1. Refer to [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration: @@ -136,6 +136,8 @@ After you have created an application, follow the [Microsoft Quickstart document - `openid` - `profile` +Alternatively, add the `User.Read.All` application permission. + ### Configuring GitLab 1. On your GitLab server, open the configuration file. @@ -154,7 +156,7 @@ After you have created an application, follow the [Microsoft Quickstart document sudo -u git -H editor config/gitlab.yml ``` -1. Refer to [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) +1. Refer to [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration: diff --git a/doc/integration/cas.md b/doc/integration/cas.md index 471b1e4e262..4699f7147aa 100644 --- a/doc/integration/cas.md +++ b/doc/integration/cas.md @@ -28,7 +28,7 @@ configure CAS for back-channel logout. sudo -u git -H editor config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration: diff --git a/doc/integration/ding_talk.md b/doc/integration/ding_talk.md new file mode 100644 index 00000000000..55d51a5ebff --- /dev/null +++ b/doc/integration/ding_talk.md @@ -0,0 +1,84 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# DingTalk OAuth 2.0 OmniAuth provider **(FREE SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/341898) in GitLab 14.5. + +You can sign in to GitLab using your DingTalk account. +Sign in to DingTalk Open Platform and create an application on it. DingTalk generates a client ID and secret key for you to use. + +1. Sign in to [DingTalk Open Platform](https://open-dev.dingtalk.com/). + +1. On the top bar, select **Application development > Enterprise internal development** and then select **Create Application**. + +  + +1. Fill in the application details: + + - **Application Name**: This can be anything. Consider something like `<Organization>'s GitLab`, or `<Your Name>'s GitLab`, or something else descriptive. + - **Application Description**: Create a description. + - **Application icon**: Upload qualified icons if needed. + +  + +1. Select **Confirm and create**. + +1. On the left sidebar, select **DingTalk Application** and find your application. Select it and go to the application information page. + +  + +1. Under the **Application Credentials** section, there should be an AppKey and AppSecret (see the screenshot). Keep this page open as you continue the configuration. + +  + +1. On your GitLab server, open the configuration file. + + For Omnibus package: + + ```shell + sudo editor /etc/gitlab/gitlab.rb + ``` + + For installations from source: + + ```shell + cd /home/git/gitlab + + sudo -u git -H editor config/gitlab.yml + ``` + +1. See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. + +1. Add the provider configuration: + + For Omnibus package: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { + "name" => "ding_talk", + "app_id" => "YOUR_APP_ID", + "app_secret" => "YOUR_APP_SECRET" + } + ] + ``` + + For installations from source: + + ```yaml + - { name: 'ding_talk', + app_id: 'YOUR_APP_ID', + app_secret: 'YOUR_APP_SECRET' } + ``` + +1. Change `YOUR_APP_ID` to the AppKey from the application information page in step 6. + +1. Change `YOUR_APP_SECRET` to the AppSecret from the application information page in step 6. + +1. Save the configuration file. + +1. [Reconfigure](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) or [restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. diff --git a/doc/integration/elasticsearch.md b/doc/integration/elasticsearch.md index b5b09fcd813..20f8fdc55f2 100644 --- a/doc/integration/elasticsearch.md +++ b/doc/integration/elasticsearch.md @@ -93,7 +93,7 @@ the indexer itself. This project relies on [International Components for Unicode](https://icu.unicode.org/) (ICU) for text encoding, therefore we must ensure the development packages for your platform are -installed before running `make`. +installed before running `make`. #### Debian / Ubuntu @@ -113,6 +113,9 @@ sudo yum install libicu-devel #### macOS +NOTE: +You must first [install Homebrew](https://brew.sh/). + To install on macOS, run: ```shell diff --git a/doc/integration/facebook.md b/doc/integration/facebook.md index 58c53db7996..1a3360aa470 100644 --- a/doc/integration/facebook.md +++ b/doc/integration/facebook.md @@ -72,7 +72,7 @@ Facebook. Facebook generates an app ID and secret key for you to use. sudo -u git -H editor config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration: diff --git a/doc/integration/github.md b/doc/integration/github.md index 16241c9293b..11a9a5ea64d 100644 --- a/doc/integration/github.md +++ b/doc/integration/github.md @@ -29,7 +29,7 @@ When you create an OAuth 2 app in GitHub, you need the following information: - The URL of your GitLab instance, such as `https://gitlab.example.com`. - The authorization callback URL; in this case, `https://gitlab.example.com/users/auth`. Include the port number if your GitLab instance uses a non-default port. -See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. After you have configured the GitHub provider, you need the following information. You must substitute that information in the GitLab configuration file in these next steps. diff --git a/doc/integration/gitlab.md b/doc/integration/gitlab.md index 5e28765bb86..b69147b3829 100644 --- a/doc/integration/gitlab.md +++ b/doc/integration/gitlab.md @@ -45,7 +45,7 @@ GitLab.com generates an application ID and secret key for you to use. sudo -u git -H editor config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration: For Omnibus installations authenticating against **GitLab.com**: diff --git a/doc/integration/gmail_action_buttons_for_gitlab.md b/doc/integration/gmail_action_buttons_for_gitlab.md index 0468e5d0a42..8b984122c8b 100644 --- a/doc/integration/gmail_action_buttons_for_gitlab.md +++ b/doc/integration/gmail_action_buttons_for_gitlab.md @@ -10,7 +10,7 @@ GitLab supports [Google actions in email](https://developers.google.com/gmail/ma If correctly set up, emails that require an action are marked in Gmail. - + To get this functioning, you must be registered with Google. For instructions, see [Register with Google](https://developers.google.com/gmail/markup/registering-with-google). diff --git a/doc/integration/google.md b/doc/integration/google.md index 172015f7528..5df76ebb3d1 100644 --- a/doc/integration/google.md +++ b/doc/integration/google.md @@ -71,7 +71,7 @@ On your GitLab server: sudo -u git -H editor config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration: For Omnibus GitLab: diff --git a/doc/integration/img/ding_talk_create_application.png b/doc/integration/img/ding_talk_create_application.png Binary files differnew file mode 100644 index 00000000000..7e58dae8299 --- /dev/null +++ b/doc/integration/img/ding_talk_create_application.png diff --git a/doc/integration/img/ding_talk_credentials.png b/doc/integration/img/ding_talk_credentials.png Binary files differnew file mode 100644 index 00000000000..f9fef7f9e51 --- /dev/null +++ b/doc/integration/img/ding_talk_credentials.png diff --git a/doc/integration/img/ding_talk_menu.png b/doc/integration/img/ding_talk_menu.png Binary files differnew file mode 100644 index 00000000000..2c5a23435fa --- /dev/null +++ b/doc/integration/img/ding_talk_menu.png diff --git a/doc/integration/img/ding_talk_your_application.png b/doc/integration/img/ding_talk_your_application.png Binary files differnew file mode 100644 index 00000000000..0864b54cc43 --- /dev/null +++ b/doc/integration/img/ding_talk_your_application.png diff --git a/doc/integration/img/enabled-oauth-sign-in-sources_v13_10.png b/doc/integration/img/enabled-oauth-sign-in-sources_v13_10.png Binary files differdeleted file mode 100644 index 86f6402c168..00000000000 --- a/doc/integration/img/enabled-oauth-sign-in-sources_v13_10.png +++ /dev/null diff --git a/doc/integration/jira/connect-app.md b/doc/integration/jira/connect-app.md index d274710b3cd..27f482ee2ba 100644 --- a/doc/integration/jira/connect-app.md +++ b/doc/integration/jira/connect-app.md @@ -6,34 +6,39 @@ info: To determine the technical writer assigned to the Stage/Group associated w # GitLab.com for Jira Cloud app **(FREE)** +You can integrate GitLab and Jira Cloud using the +[GitLab.com for Jira Cloud](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud) +app in the Atlassian Marketplace. + NOTE: -Only Jira users with administrator level access are able to install or configure +Only Jira users with the administrator role can install or configure the GitLab.com for Jira Cloud app. -## GitLab.com for Jira Cloud app **(FREE SAAS)** +## Install the GitLab.com for Jira Cloud app **(FREE SAAS)** -You can integrate GitLab.com and Jira Cloud using the -[GitLab.com for Jira Cloud](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud) -app in the Atlassian Marketplace. The user configuring GitLab.com for Jira Cloud app must have -[Maintainer](../../user/permissions.md) permissions in the GitLab.com namespace. +If you use GitLab.com and Jira Cloud, you can install the GitLab.com for Jira Cloud app. +If you do not use both of these environments, use the [Jira DVCS Connector](dvcs.md) or +[install GitLab.com for Jira Cloud app for self-managed instances](#install-the-gitlabcom-for-jira-cloud-app-for-self-managed-instances). +We recommend the GitLab.com for Jira Cloud app, because data is +synchronized in real time. The DVCS connector updates data only once per hour. -This integration method supports [smart commits](dvcs.md#smart-commits). +The user configuring the GitLab.com for Jira Cloud app must have +at least the [Maintainer](../../user/permissions.md) role in the GitLab.com namespace. -This method is recommended when using GitLab.com and Jira Cloud because data is -synchronized in real-time. The DVCS connector updates data only once per hour. -If you are not using both of these environments, use the [Jira DVCS Connector](dvcs.md) method or -[steps to install GitLab.com for Jira Cloud app for self-managed instances](#install-the-gitlabcom-for-jira-cloud-app-for-self-managed-instances). +This integration method supports [Smart Commits](dvcs.md#smart-commits). <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> For a walkthrough of the integration with GitLab.com for Jira Cloud app, watch [Configure GitLab.com Jira Could Integration using Marketplace App](https://youtu.be/SwR-g1s1zTo) on YouTube. -1. Go to **Jira Settings > Apps > Find new apps**, then search for GitLab. -1. Click **GitLab.com for Jira Cloud**, then click **Get it now**, or go to the +To install the GitLab.com for Jira Cloud app: + +1. In Jira, go to **Jira Settings > Apps > Find new apps**, then search for GitLab. +1. Select **GitLab.com for Jira Cloud**, then select **Get it now**, or go to the [App in the marketplace directly](https://marketplace.atlassian.com/apps/1221011/gitlab-com-for-jira-cloud).  -1. After installing, click **Get started** to go to the configurations page. +1. After installing, to go to the configurations page, select **Get started**. This page is always available under **Jira Settings > Apps > Manage apps**.  @@ -41,7 +46,7 @@ For a walkthrough of the integration with GitLab.com for Jira Cloud app, watch [Maintainer](../../user/permissions.md) permissions to add namespaces.  -1. Select **Add namespace** to open the list of available namespaces. +1. To open the list of available namespaces, select **Add namespace**. 1. Identify the namespace you want to link, and select **Link**. Only Jira site administrators are permitted to add or remove namespaces for an installation. @@ -89,30 +94,30 @@ from outside the Marketplace, which allows you to install the application: 1. Sign in to your Jira instance as a user with an Administrator role. 1. Place your Jira instance into [development mode](https://developer.atlassian.com/cloud/jira/platform/getting-started-with-connect/#step-2--enable-development-mode). -1. Sign in to your GitLab application as a user with an [Administrator](../../user/permissions.md) role. +1. Sign in to your GitLab application as an [administrator](../../user/permissions.md). 1. Install the GitLab application from your self-managed GitLab instance, as described in the [Atlassian developer guides](https://developer.atlassian.com/cloud/jira/platform/getting-started-with-connect/#step-3--install-and-test-your-app): - 1. In your Jira instance, go to **Apps > Manage Apps** and click **Upload app**: + 1. In your Jira instance, go to **Apps > Manage Apps** and select **Upload app**: -  +  - 1. For **App descriptor URL**, provide full URL to your manifest file, modifying this - URL based on your instance configuration: `https://your.domain/your-path/-/jira_connect/app_descriptor.json` - 1. Click **Upload**, and Jira fetches the content of your `app_descriptor` file and installs - it for you. + 1. For **App descriptor URL**, provide the full URL to your manifest file, based + on your instance configuration. For example: `https://your.domain/your-path/-/jira_connect/app_descriptor.json`. + 1. Select **Upload**. Jira fetches the content of your `app_descriptor` file and installs + it. 1. If the upload is successful, Jira displays a modal panel: **Installed and ready to go!** - Click **Get started** to configure the integration. + To configure the integration, select **Get started**. -  +  1. Disable [development mode](https://developer.atlassian.com/cloud/jira/platform/getting-started-with-connect/#step-2--enable-development-mode) on your Jira instance. The **GitLab.com for Jira Cloud** app now displays under **Manage apps**. You can also -click **Get started** to open the configuration page rendered from your GitLab instance. +select **Get started** to open the configuration page rendered from your GitLab instance. NOTE: -If a GitLab update makes changes to the application descriptor, you must uninstall, then reinstall, the -application. +If a GitLab update makes changes to the application descriptor, you must uninstall, +then reinstall the application. ### Create a Marketplace listing @@ -120,31 +125,33 @@ If you prefer to not use development mode on your Jira instance, you can create your own Marketplace listing for your instance. This enables your application to be installed from the Atlassian Marketplace. -For full instructions, review the Atlassian [guide to creating a marketplace listing](https://developer.atlassian.com/platform/marketplace/installing-cloud-apps/#creating-the-marketplace-listing). To create a -Marketplace listing, you must: +For full instructions, review the Atlassian [guide to creating a marketplace listing](https://developer.atlassian.com/platform/marketplace/installing-cloud-apps/#creating-the-marketplace-listing). +To create a Marketplace listing: 1. Register as a Marketplace vendor. -1. List your application, using the application descriptor URL. +1. List your application using the application descriptor URL. - Your manifest file is located at: `https://your.domain/your-path/-/jira_connect/app_descriptor.json` - - GitLab recommends you list your application as `private`, because public + - We recommend you list your application as `private`, because public applications can be viewed and installed by any user. 1. Generate test license tokens for your application. -Review the -[official Atlassian documentation](https://developer.atlassian.com/platform/marketplace/installing-cloud-apps/#creating-the-marketplace-listing) -for details. - NOTE: -Using this method, [updates are automated](#update-the-gitlabcom-for-jira-cloud-app) -the same way as when using our GitLab.com Marketplace listing. +This method uses [automated updates](#update-the-gitlabcom-for-jira-cloud-app) +the same way as our GitLab.com Marketplace listing. ## Troubleshoot GitLab.com for Jira Cloud app -The GitLab.com for Jira Cloud app uses an iframe to add namespaces on the -settings page. Some browsers block cross-site cookies, which can lead to a -message saying that the user needs to log in on GitLab.com even though the user -is already logged in. +### Browser displays sign-in message when already signed in + +You might get the following message prompting you to sign in to GitLab.com +when you're already signed in: + +```plaintext +You need to sign in or sign up before continuing. +``` -> "You need to sign in or sign up before continuing." +GitLab.com for Jira Cloud app uses an iframe to add namespaces on the +settings page. Some browsers block cross-site cookies, which can lead to this issue. -In this case, use [Firefox](https://www.mozilla.org/en-US/firefox/), [Google Chrome](https://www.google.com/chrome/), or enable cross-site cookies in your browser. +To resolve this issue, use [Firefox](https://www.mozilla.org/en-US/firefox/), +[Google Chrome](https://www.google.com/chrome/), or enable cross-site cookies in your browser. diff --git a/doc/integration/jira/development_panel.md b/doc/integration/jira/development_panel.md index 6fa084ee872..8f66edcffa8 100644 --- a/doc/integration/jira/development_panel.md +++ b/doc/integration/jira/development_panel.md @@ -38,7 +38,7 @@ After the integration is [set up on GitLab and Jira](#configure-the-integration) - Refer to any Jira issue by its ID (in uppercase) in GitLab branch names, commit messages, and merge request titles. - See the linked branches, commits, and merge requests in Jira issues. -- Create GitLab branches from Jira issues ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66032) in GitLab 14.2). +- Create GitLab branches from Jira Cloud issues ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66032) in GitLab 14.2). At this time, merge requests are called "pull requests" in Jira issues. This name may change in a future Jira release. @@ -89,7 +89,10 @@ This integration is not supported on GitLab instances under a [relative URL](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-a-relative-url-for-gitlab). For example, `http://example.com/gitlab`. -## Troubleshooting +## Troubleshoot the Development Panel + +If you use Jira on your own server, go to the [Atlassian documentation](https://confluence.atlassian.com/jirakb/troubleshoot-the-development-panel-in-jira-server-574685212.html) +for general troubleshooting information. ### Cookies for Oracle's Access Manager diff --git a/doc/integration/jira/dvcs.md b/doc/integration/jira/dvcs.md index 664a0361da4..2b7cc5ff0a6 100644 --- a/doc/integration/jira/dvcs.md +++ b/doc/integration/jira/dvcs.md @@ -19,7 +19,7 @@ are accessible. - **Jira Server**: Your network must allow access to your instance. - **Jira Cloud**: Your instance must be accessible through the internet. -## Smart commits +## Smart Commits When connecting GitLab with Jira with DVCS, you can process your Jira issues using special commands, called @@ -48,17 +48,24 @@ Smart Commits should follow the pattern of: Some examples: -- Adding a comment to a Jira issue: `KEY-123 fixes a bug #comment Bug is fixed.` -- Recording time tracking: `KEY-123 #time 2w 4d 10h 52m Tracking work time.` -- Closing an issue: `KEY-123 #close Closing issue` +- Add a comment to a Jira issue: `KEY-123 fixes a bug #comment Bug is fixed.` +- Record time tracking: `KEY-123 #time 2w 4d 10h 52m Tracking work time.` +- Close an issue: `KEY-123 #close Closing issue` A Smart Commit message must not span more than one line (no carriage returns) but -you can still perform multiple actions in a single commit: +you can still perform multiple actions in a single commit. For example: -- Time tracking, commenting, and transitioning to **Closed**: - `KEY-123 #time 2d 5h #comment Task completed ahead of schedule #close`. -- Commenting, transitioning to **In-progress**, and time tracking: - `KEY-123 #comment started working on the issue #in-progress #time 12d 5h`. +- Add time tracking, add a comment, and transition to **Closed**: + + ```plaintext + KEY-123 #time 2d 5h #comment Task completed ahead of schedule #close + ``` + +- Add a comment, transition to **In-progress**, and add time tracking: + + ```plaintext + KEY-123 #comment started working on the issue #in-progress #time 12d 5h + ``` ## Configure a GitLab application for DVCS @@ -69,9 +76,9 @@ you can set up this integration with your own account instead. 1. In GitLab, [create a user](../../user/profile/account/create_accounts.md) for Jira to use to connect to GitLab. This user must be added to each project you want Jira to have access to, - or have an [Administrator](../../user/permissions.md) role to access all projects. + or be an administrator to access all projects. 1. Sign in as the `jira` user. -1. In the top right corner, click the account's avatar, and select **Edit profile**. +1. On the top bar, in the top right corner, select the user's avatar, and select **Edit profile**. 1. On the left sidebar, select **Applications**. 1. In the **Name** field, enter a descriptive name for the integration, such as `Jira`. 1. In the **Redirect URI** field, enter the URI appropriate for your version of GitLab, @@ -86,9 +93,10 @@ you can set up this integration with your own account instead. `https://<gitlab.example.com>/-/jira/login/oauth/callback`. 1. For **Scopes**, select `api` and clear any other checkboxes. + - The DVCS connector requires a _write-enabled_ `api` scope to automatically create and manage required webhooks. 1. Select **Submit**. -1. GitLab displays the generated **Application ID** - and **Secret** values. Copy these values, as you need them to configure Jira. +1. Copy the **Application ID** and **Secret** values. + You need them to configure Jira. ## Configure Jira for DVCS @@ -96,19 +104,21 @@ Configure this connection when you want to import all GitLab commits and branche for the groups you specify, into Jira. This import takes a few minutes and, after it completes, refreshes every 60 minutes: -1. Ensure you have completed the [GitLab configuration](#configure-a-gitlab-application-for-dvcs). +1. Complete the [GitLab configuration](#configure-a-gitlab-application-for-dvcs). 1. Go to your DVCS accounts: - - *For Jira Server,* go to **Settings (gear) > Applications > DVCS accounts**. - - *For Jira Cloud,* go to **Settings (gear) > Products > DVCS accounts**. + - *For Jira Server,* select **Settings (gear) > Applications > DVCS accounts**. + - *For Jira Cloud,* select **Settings (gear) > Products > DVCS accounts**. 1. To create a new integration, select the appropriate value for **Host**: - *For Jira versions 8.14 and later:* Select **GitLab** or **GitLab Self-Managed**. - - *For Jira versions 8.13 and earlier:* Select **GitHub Enterprise**. + - *For Jira Cloud or Jira versions 8.13 and earlier:* Select **GitHub Enterprise**. 1. For **Team or User Account**, enter either: - *For Jira versions 8.14 and later:* - - The relative path of a top-level GitLab group that [the GitLab user](#configure-a-gitlab-application-for-dvcs) has access to. - - *For Jira versions 8.13 and earlier:* - - The relative path of a top-level GitLab group that [the GitLab user](#configure-a-gitlab-application-for-dvcs) has access to. + - The relative path of a top-level GitLab group that + [the GitLab user](#configure-a-gitlab-application-for-dvcs) has access to. + - *For Jira Cloud or Jira versions 8.13 and earlier:* + - The relative path of a top-level GitLab group that + [the GitLab user](#configure-a-gitlab-application-for-dvcs) has access to. - The relative path of your personal namespace. 1. In the **Host URL** field, enter the URI appropriate for your version of GitLab, @@ -119,13 +129,13 @@ it completes, refreshes every 60 minutes: 1. For **Client ID**, use the **Application ID** value from the previous section. 1. For **Client Secret**, use the **Secret** value from the previous section. -1. Ensure that the rest of the checkboxes are checked. -1. Select **Add** and then **Continue** to create the DVCS account. -1. Jira redirects to GitLab where you have to confirm the authorization, - and then GitLab redirects back to Jira where you should see the synced - projects show up inside the new account. +1. Ensure that the rest of the checkboxes are selected. +1. To create the DVCS account, select **Add** and then **Continue**. +1. Jira redirects to GitLab where you have to confirm the authorization. + GitLab then redirects back to Jira where the synced + projects should display in the new account. -To connect additional GitLab projects from other GitLab top-level groups, or +To connect additional GitLab projects from other GitLab top-level groups or personal namespaces, repeat the previous steps with additional Jira DVCS accounts. After you configure the integration, read more about [how to test and use it](development_panel.md). @@ -171,9 +181,8 @@ Error obtaining access token. Cannot access https://gitlab.example.com from Jira as GitLab is the TLS client. - The Jira Development panel integration requires Jira to connect to GitLab, which causes Jira to be the TLS client. If your GitLab server's certificate is not - issued by a public certificate authority, the Java Truststore on Jira's server - must have the appropriate certificate (such as your organization's - root certificate) added to it . + issued by a public certificate authority, add the appropriate certificate + (such as your organization's root certificate) to the Java Truststore on Jira's server. Refer to Atlassian's documentation and Atlassian Support for assistance setting up Jira correctly: @@ -186,8 +195,8 @@ up Jira correctly: - If the integration stops working after upgrading Jira's Java runtime, the `cacerts` Truststore may have been replaced during the upgrade. -- Troubleshooting connectivity [up to and including TLS handshaking](https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html), - using the a java class called `SSLPoke`. +- Troubleshoot connectivity [up to and including TLS handshaking](https://confluence.atlassian.com/kb/unable-to-connect-to-ssl-services-due-to-pkix-path-building-failed-error-779355358.html), + using the `SSLPoke` Java class. - Download the class from Atlassian's knowledge base to a directory on Jira's server, such as `/tmp`. - Use the same Java runtime as Jira. - Pass all networking-related parameters that Jira is called with, such as proxy @@ -202,7 +211,7 @@ The message `Successfully connected` indicates a successful TLS handshake. If there are problems, the Java TLS library generates errors that you can look up for more detail. -### Scope error when connecting Jira via DVCS +### Scope error when connecting to Jira using DVCS ```plaintext The requested scope is invalid, unknown, or malformed. @@ -223,12 +232,12 @@ After you complete the **Add New Account** form in Jira and authorize access, yo encounter these issues: - An `Error! Failed adding the account: [Error retrieving list of repositories]` error. -- An `Account is already integrated with JIRA` error when you click **Try Again**. +- An `Account is already integrated with JIRA` error when you select **Try Again**. - An account is visible in the DVCS accounts view, but no repositories are listed. To resolve this issue: -- If you're using GitLab Free, be sure you're using GitLab 13.4 or later. +- If you're using GitLab Free, ensure you're using GitLab 13.4 or later. - If you're using GitLab versions 11.10-12.7, upgrade to GitLab 12.8.10 or later to resolve [an identified issue](https://gitlab.com/gitlab-org/gitlab/-/issues/37012). @@ -242,17 +251,17 @@ This issue occurs when you use the Jira DVCS connector and your integration is c For more information and possible fixes, see [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340160). -### Fix synchronization issues +### Synchronization issues If Jira displays incorrect information, such as deleted branches, you may have to -resynchronize the information. To do so: - -1. In Jira, go to **Jira Administration > Applications > DVCS accounts**. -1. At the account (group or subgroup) level, Jira displays an option to - **Refresh repositories** in the **{ellipsis_h}** (ellipsis) menu. -1. For each project, there's a sync button displayed next to the **last activity** date. - - To perform a *soft resync*, click the button. - - To complete a *full sync*, shift-click the button. +resynchronize the information: + +1. In Jira, select **Jira Administration > Applications > DVCS accounts**. +1. For the account (group or subgroup), select + **Refresh repositories** from the **{ellipsis_h}** (ellipsis) menu. +1. For each project, next to the **Last activity** date: + - To perform a *soft resync*, select the sync icon. + - To complete a *full sync*, press `Shift` and select the sync icon. For more information, read [Atlassian's documentation](https://support.atlassian.com/jira-cloud-administration/docs/synchronize-jira-cloud-to-bitbucket/). diff --git a/doc/integration/jira/index.md b/doc/integration/jira/index.md index f5f7e8c33fc..3052d85b2cb 100644 --- a/doc/integration/jira/index.md +++ b/doc/integration/jira/index.md @@ -10,7 +10,7 @@ If your organization uses [Jira](https://www.atlassian.com/software/jira) issues you can [migrate your issues from Jira](../../user/project/import/jira.md) and work exclusively in GitLab. However, if you'd like to continue to use Jira, you can integrate it with GitLab. GitLab offers two types of Jira integrations, and you -can use one or both depending on the capabilities you need. It is recommended that you enable both. +can use one or both depending on the capabilities you need. We recommend you enable both. ## Compare integrations @@ -41,7 +41,7 @@ or the Jira DVCS (distributed version control system) connector, ### Direct feature comparison -| Capability | Jira integration | Jira Development panel integration | +| Capability | Jira integration | Jira development panel integration | |-|-|-| | Mention a Jira issue ID in a GitLab commit or merge request, and a link to the Jira issue is created. | Yes. | No. | | Mention a Jira issue ID in GitLab and the Jira issue shows the GitLab issue or merge request. | Yes. A Jira comment with the GitLab issue or MR title links to GitLab. The first mention is also added to the Jira issue under **Web links**. | Yes, in the issue's [development panel](https://support.atlassian.com/jira-software-cloud/docs/view-development-information-for-an-issue/). | @@ -55,15 +55,15 @@ or the Jira DVCS (distributed version control system) connector, ## Authentication in Jira -The process for configuring Jira depends on whether you host Jira on your own server or on +The authentication method in Jira depends on whether you host Jira on your own server or on [Atlassian cloud](https://www.atlassian.com/cloud): - **Jira Server** supports basic authentication. When connecting, a **username and password** are - required. Connecting to Jira Server via CAS is not possible. For more information, read + required. Connecting to Jira Server using the Central Authentication Service (CAS) is not possible. For more information, read how to [set up a user in Jira Server](jira_server_configuration.md). - **Jira on Atlassian cloud** supports authentication through an API token. When connecting to Jira on Atlassian cloud, an email and API token are required. For more information, read - [set up a user in Jira on Atlassian cloud](jira_cloud_configuration.md). + [create an API token for Jira in Atlassian cloud](jira_cloud_configuration.md). ## Privacy considerations @@ -72,11 +72,16 @@ actions in GitLab issues and merge requests linked to a Jira issue leak informat about the private project to non-administrator Jira users. If your installation uses Jira Cloud, you can use the [GitLab.com for Jira Cloud app](connect-app.md) to avoid this risk. +## Third-party Jira integrations + +Developers have built several third-party Jira integrations for GitLab that are +listed on the [Atlassian Marketplace](https://marketplace.atlassian.com/search?product=jira&query=gitlab). + ## Troubleshooting If these features do not work as expected, it is likely due to a problem with the way the integration settings were configured. -### GitLab is unable to comment on a Jira issue +### GitLab cannot comment on a Jira issue If GitLab cannot comment on Jira issues, make sure the Jira user you set up for the integration has permission to: @@ -86,14 +91,16 @@ set up for the integration has permission to: Jira issue references and update comments do not work if the GitLab issue tracker is disabled. -### GitLab is unable to close a Jira issue +### GitLab cannot close a Jira issue + +If GitLab cannot close a Jira issue: -Make sure the `Transition ID` you set in the Jira settings matches the one -your project needs to close an issue. +- Make sure the `Transition ID` you set in the Jira settings matches the one + your project needs to close an issue. -Make sure that the Jira issue is not already marked as resolved. That is, -the Jira issue resolution field is not set, and the issue is not struck through in -Jira lists. +- Make sure the Jira issue is not already marked as resolved: + - Check the Jira issue resolution field is not set. + - Check the issue is not struck through in Jira lists. ### CAPTCHA @@ -104,8 +111,3 @@ authenticate with the Jira site. To fix this error, sign in to your Jira instance and complete the CAPTCHA. - -## Third-party Jira integrations - -Developers have built several third-party Jira integrations for GitLab that are -listed on the [Atlassian Marketplace](https://marketplace.atlassian.com/search?product=jira&query=gitlab). diff --git a/doc/integration/jira/jira_cloud_configuration.md b/doc/integration/jira/jira_cloud_configuration.md index 0cfffdb8ba4..08cd34860ff 100644 --- a/doc/integration/jira/jira_cloud_configuration.md +++ b/doc/integration/jira/jira_cloud_configuration.md @@ -4,18 +4,19 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Create an API token in Jira on Atlassian cloud **(FREE)** +# Create an API token for Jira in Atlassian cloud **(FREE)** You need an API token to [integrate with Jira](index.md) on Atlassian cloud. To create the API token: -1. Sign in to [`id.atlassian.com`](https://id.atlassian.com/manage-profile/security/api-tokens) - with your email address. Use an account with *write* access to Jira projects. -1. Go to **Settings > Atlassian account settings > Security > Create and manage API tokens**. -1. Select **Create API token** to display a modal window with an API token. +1. Sign in to [Atlassian](https://id.atlassian.com/manage-profile/security/api-tokens) + using an account with *write* access to Jira projects. + + The link opens the API tokens page. Alternatively, to go to this page from your Atlassian + profile, select **Account Settings > Security > Create and manage API tokens**. +1. Select **Create API token**. 1. In the dialog, enter a label for your token and select **Create**. -1. To copy the API token, select **Copy**, then paste the token somewhere safe. You need this value when you - [configure GitLab](configure.md). +1. To copy the API token, select **Copy**, then paste the token somewhere safe. You need the newly created token, and the email address you used when you created it, when you diff --git a/doc/integration/jira/jira_server_configuration.md b/doc/integration/jira/jira_server_configuration.md index 32a8cd430f9..63625dd5065 100644 --- a/doc/integration/jira/jira_server_configuration.md +++ b/doc/integration/jira/jira_server_configuration.md @@ -17,9 +17,9 @@ credentials, you must: ## Create a Jira Server user -This process creates a user named `gitlab` and adds it to a new group named `gitlab-developers`: +This process creates a user named `gitlab`: -1. Sign in to your Jira instance as an administrator. +1. Sign in to your Jira instance as a Jira administrator. 1. In the upper right corner of the top menu, go to the gear icon and select **User Management**. 1. Create a new user account (`gitlab`) with write access to @@ -37,13 +37,15 @@ After you create the user, create a group for it. ## Create a Jira Server group -After you [create a Jira Server user](#create-a-jira-server-user), you can create a -group to assign permissions to the user: +After you [create a Jira Server user](#create-a-jira-server-user), create a +group to assign permissions to the user. -1. Sign in to your Jira instance as an administrator. +This process adds the `gitlab` user you created to a new group named `gitlab-developers`: + +1. Sign in to your Jira instance as a Jira administrator. 1. In the upper right corner of the top menu, go to the gear icon and select **User Management**. -1. From the sidebar, select **Groups**. +1. On the sidebar, select **Groups**.  @@ -52,12 +54,12 @@ group to assign permissions to the user: 1. To add the `gitlab` user to the `gitlab-developers` group, select **Edit members**. The `gitlab-developers` group should be listed in the leftmost box as a selected group. -1. In the **Add members to selected group(s)** area, enter `gitlab`. +1. In the **Add members to selected group(s)** section, enter `gitlab`. 1. Select **Add selected users**. -Jira saves your selection, and `gitlab` should appear in the **Group member(s)** -area. + The `gitlab` user appears in the **Group member(s)** + section. - +  Next, create a permission scheme for your group. @@ -65,16 +67,16 @@ Next, create a permission scheme for your group. After creating the group in Jira, grant permissions to the group by creating a permission scheme: -1. Sign in to your Jira instance as an administrator. +1. Sign in to your Jira instance as a Jira administrator. 1. In the upper right corner of the top menu, go to the gear icon and select **Issues**. -1. From the sidebar, select **Permission Schemes**. +1. On the sidebar, select **Permission Schemes**. 1. Select **Add Permission Scheme**, enter a **Name** and (optionally) a **Description**, and then select **Add**. 1. In the permissions scheme list, locate your new permissions scheme, and select **Permissions**. -1. Next to **Administer Projects**, select **Edit**. In - the **Group** list, select `gitlab-developers`. +1. Next to **Administer Projects**, select **Edit**. +1. From the **Group** dropdown list, select `gitlab-developers`, and then select **Grant**.  diff --git a/doc/integration/kerberos.md b/doc/integration/kerberos.md index 2d4abb75875..0f9bf3ba1d1 100644 --- a/doc/integration/kerberos.md +++ b/doc/integration/kerberos.md @@ -100,7 +100,7 @@ to authenticate with Kerberos tokens. #### Enable single sign-on -See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) +See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings to enable single sign-on and add Kerberos servers as an identity provider. @@ -137,7 +137,7 @@ with your Kerberos credentials. The first time users sign in to GitLab with their Kerberos accounts, GitLab creates a matching account. -Before you continue, review the [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) options in Omnibus and GitLab source. You must also include `kerberos`. +Before you continue, review the [Configure initial settings](omniauth.md#configure-initial-settings) options in Omnibus and GitLab source. You must also include `kerberos`. With that information at hand: diff --git a/doc/integration/mattermost/index.md b/doc/integration/mattermost/index.md index 39005940dfc..0489ccd431c 100644 --- a/doc/integration/mattermost/index.md +++ b/doc/integration/mattermost/index.md @@ -1,7 +1,7 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # GitLab Mattermost @@ -428,7 +428,7 @@ mattermost['env'] = { Refer to the [Mattermost Configuration Settings documentation](https://docs.mattermost.com/administration/config-settings.html) -for details about categories, configuration values, etc. +for details about categories and configuration values. There are a few exceptions to this rule: diff --git a/doc/integration/oauth2_generic.md b/doc/integration/oauth2_generic.md index bdf6a0e687d..cdc7e6db61c 100644 --- a/doc/integration/oauth2_generic.md +++ b/doc/integration/oauth2_generic.md @@ -55,9 +55,42 @@ This strategy is designed to allow configuration of the simple OmniAuth SSO proc sudo -u git -H editor config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings +1. See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings + +1. Add the provider-specific configuration for your provider, for example: + + ```ruby + gitlab_rails['omniauth_providers'] = [ + { 'name' => 'oauth2_generic', + 'label' => '<your_oauth2_label>', + 'app_id' => '<your_app_client_id>', + 'app_secret' => '<your_app_client_secret>', + 'args' => { + client_options: { + 'site' => '<your_auth_server_url>', + 'user_info_url' => '/oauth2/v1/userinfo', + 'authorize_url' => '/oauth2/v1/authorize', + 'token_url' => '/oauth2/v1/token' + }, + user_response_structure: { + root_path: [], + id_path: ['sub'], + attributes: { + email: 'email', + name: 'name' + } + }, + authorize_params: { + scope: 'openid profile email' + }, + strategy_class: "OmniAuth::Strategies::OAuth2Generic" + } + } + } + ] + ``` -1. Add the provider-specific configuration for your provider, as [described in the gem's README](https://gitlab.com/satorix/omniauth-oauth2-generic#gitlab-config-example) + For more information about these settings, see [the gem's README](https://gitlab.com/satorix/omniauth-oauth2-generic#gitlab-config-example). 1. Save the configuration file diff --git a/doc/integration/omniauth.md b/doc/integration/omniauth.md index 6c154ee7f5b..5e96a1e7c65 100644 --- a/doc/integration/omniauth.md +++ b/doc/integration/omniauth.md @@ -6,25 +6,16 @@ info: To determine the technical writer assigned to the Stage/Group associated w # OmniAuth **(FREE SELF)** -GitLab leverages OmniAuth to allow users to sign in using Twitter, GitHub, and -other popular services. [OmniAuth](https://rubygems.org/gems/omniauth/) is a -"generalized Rack framework for multiple-provider authentication" built on Ruby. +Users can sign in to GitLab by using their credentials from Twitter, GitHub, and other popular services. +[OmniAuth](https://rubygems.org/gems/omniauth/) is the Rack +framework that GitLab uses to provide this authentication. -Configuring OmniAuth does not prevent standard GitLab authentication or LDAP -(if configured) from continuing to work. Users can choose to sign in using any -of the configured mechanisms. +If you configure OmniAuth, users can continue to sign in using other +mechanisms, including standard GitLab authentication or LDAP (if configured). -- [Initial OmniAuth Configuration](#initial-omniauth-configuration) -- [Supported Providers](#supported-providers) -- [Enable OmniAuth for an Existing User](#enable-omniauth-for-an-existing-user) -- [OmniAuth configuration sample when using Omnibus GitLab](https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master#omniauth-google-twitter-github-login) -- [Enable or disable Sign In with an OmniAuth provider without disabling import sources](#enable-or-disable-sign-in-with-an-omniauth-provider-without-disabling-import-sources) +## Supported providers -## Supported Providers - -This is a list of the current supported OmniAuth providers. Before proceeding on -each provider's documentation, make sure to first read this document as it -contains some settings that are common for all providers. +GitLab supports the following OmniAuth providers. | Provider documentation | OmniAuth provider name | |---------------------------------------------------------------------|----------------------------| @@ -37,6 +28,7 @@ contains some settings that are common for all providers. | [Azure v1](azure.md) | `azure_oauth2` | | [Bitbucket Cloud](bitbucket.md) | `bitbucket` | | [CAS](cas.md) | `cas3` | +| [DingTalk](ding_talk.md) | `ding_talk` | | [Facebook](facebook.md) | `facebook` | | [Generic OAuth 2.0](oauth2_generic.md) | `oauth2_generic` | | [GitHub](github.md) | `github` | @@ -50,127 +42,101 @@ contains some settings that are common for all providers. | [Shibboleth](saml.md) | `shibboleth` | | [Twitter](twitter.md) | `twitter` | -## Initial OmniAuth Configuration - -The OmniAuth provider names from the table above are needed to configure a few -global settings that are in common for all providers. +## Configure initial settings NOTE: -Starting from GitLab 11.4, OmniAuth is enabled by default. If you're using an +In GitLab 11.4 and later, OmniAuth is enabled by default. If you're using an earlier version, you must explicitly enable it. -- `allow_single_sign_on` allows you to specify the providers that automatically - create a GitLab account. For example, if you wish to enable Azure (v2) and Google, - in Omnibus, specify a list of provider names: - - ```ruby - gitlab_rails['omniauth_allow_single_sign_on'] = ['azure_activedirectory_v2', 'google_oauth2'] - ``` - - The value defaults to `false`. If `false` users must be created manually, or - they can't sign in by using OmniAuth. - -- `auto_link_ldap_user` can be used if you have [LDAP / ActiveDirectory](../administration/auth/ldap/index.md) - integration enabled. It defaults to `false`. When enabled, users automatically - created through an OmniAuth provider have their LDAP identity created in GitLab as well. -- `block_auto_created_users` defaults to `true`. If `true`, auto created users will - be blocked pending approval by an administrator before they are able to sign in. - -NOTE: -If you set `block_auto_created_users` to `false`, make sure to only -define providers under `allow_single_sign_on` that you are able to control, like -SAML, Shibboleth, Crowd, or Google. Otherwise, set it to `true`, or any user on -the Internet can successfully sign in to your GitLab without -administrative approval. +Before you configure the OmniAuth provider, +configure the settings that are common for all providers. -NOTE: -`auto_link_ldap_user` requires the `uid` of the user to be the same in both LDAP -and the OmniAuth provider. +Setting | Description | Default value +---------------------------|-------------|-------------- +`allow_single_sign_on` | Enables you to list the providers that automatically create a GitLab account. The provider names are available in the **OmniAuth provider name** column in the [supported providers table](#supported-providers). | The default is `false`. If `false`, users must be created manually, or they can't sign in using OmniAuth. +`auto_link_ldap_user` | If enabled, creates an LDAP identity in GitLab for users that are created through an OmniAuth provider. You can enable this setting if you have the [LDAP (ActiveDirectory)](../administration/auth/ldap/index.md) integration enabled. Requires the `uid` of the user to be the same in both LDAP and the OmniAuth provider. | The default is `false`. +`block_auto_created_users` | If enabled, blocks users that are automatically created from signing in until they are approved by an administrator. | The default is `true`. If you set the value to `false`, make sure you only define providers for `allow_single_sign_on` that you can control, like SAML, Shibboleth, Crowd, or Google. Otherwise, any user on the internet can sign in to GitLab without an administrator's approval. To change these settings: - **For Omnibus package** - Open the configuration file: + 1. Open the configuration file: - ```shell - sudo editor /etc/gitlab/gitlab.rb - ``` + ```shell + sudo editor /etc/gitlab/gitlab.rb + ``` - and change: + 1. Update the following section: - ```ruby - # CAUTION! - # This allows users to sign in without having a user account first. Define the allowed providers - # using an array, for example, ["saml", "twitter"], or as true/false to allow all providers or none. - # User accounts will be created automatically when authentication was successful. - gitlab_rails['omniauth_allow_single_sign_on'] = ['saml', 'twitter'] - gitlab_rails['omniauth_auto_link_ldap_user'] = true - gitlab_rails['omniauth_block_auto_created_users'] = true - ``` + ```ruby + # CAUTION! + # This allows users to sign in without having a user account first. Define the allowed providers + # using an array, for example, ["saml", "twitter"], or as true/false to allow all providers or none. + # User accounts will be created automatically when authentication was successful. + gitlab_rails['omniauth_allow_single_sign_on'] = ['saml', 'twitter'] + gitlab_rails['omniauth_auto_link_ldap_user'] = true + gitlab_rails['omniauth_block_auto_created_users'] = true + ``` - **For installations from source** - Open the configuration file: + 1. Open the configuration file: - ```shell - cd /home/git/gitlab + ```shell + cd /home/git/gitlab - sudo -u git -H editor config/gitlab.yml - ``` + sudo -u git -H editor config/gitlab.yml + ``` - and change the following section: + 1. Update the following section: - ```yaml - ## OmniAuth settings - omniauth: - # Allow sign-in by using Twitter, Google, etc. using OmniAuth providers - # Versions prior to 11.4 require this to be set to true - # enabled: true + ```yaml + ## OmniAuth settings + omniauth: + # Allow sign-in by using Twitter, Google, etc. using OmniAuth providers + # Versions prior to 11.4 require this to be set to true + # enabled: true - # CAUTION! - # This allows users to sign in without having a user account first. Define the allowed providers - # using an array, for example, ["saml", "twitter"], or as true/false to allow all providers or none. - # User accounts will be created automatically when authentication was successful. - allow_single_sign_on: ["saml", "twitter"] + # CAUTION! + # This allows users to sign in without having a user account first. Define the allowed providers + # using an array, for example, ["saml", "twitter"], or as true/false to allow all providers or none. + # User accounts will be created automatically when authentication was successful. + allow_single_sign_on: ["saml", "twitter"] - auto_link_ldap_user: true + auto_link_ldap_user: true - # Locks down those users until they have been cleared by the admin (default: true). - block_auto_created_users: true - ``` + # Locks down those users until they have been cleared by the admin (default: true). + block_auto_created_users: true + ``` -Now we can choose one or more of the [Supported Providers](#supported-providers) -listed above to continue the configuration process. +After configuring these settings, you can configure +your chosen [provider](#supported-providers). -## Enable OmniAuth for an Existing User +## Enable OmniAuth for an existing user -Existing users can enable OmniAuth for specific providers after the account is -created. For example, if the user originally signed in with LDAP, an OmniAuth -provider such as Twitter can be enabled. Follow the steps below to enable an -OmniAuth provider for an existing user. +If you're an existing user, after your GitLab account is +created, you can activate an OmniAuth provider. For example, if you originally signed in with LDAP, you can enable an OmniAuth +provider like Twitter. -1. Sign in normally - whether standard sign in, LDAP, or another OmniAuth provider. -1. In the top-right corner, select your avatar. +1. Sign in to GitLab with your GitLab credentials, LDAP, or another OmniAuth provider. +1. On the top bar, in the top right corner, select your avatar. 1. Select **Edit profile**. 1. On the left sidebar, select **Account**. -1. In the **Connected Accounts** section, select the desired OmniAuth provider, such as Twitter. -1. The user is redirected to the provider. After the user authorizes GitLab, - they are redirected back to GitLab. +1. In the **Connected Accounts** section, select the OmniAuth provider, such as Twitter. +1. You are redirected to the provider. After you authorize GitLab, + you are redirected back to GitLab. -The chosen OmniAuth provider is now active and can be used to sign in to GitLab from then on. +You can now use your chosen OmniAuth provider to sign in to GitLab. -## Automatically Link Existing Users to OmniAuth Users +## Link existing users to OmniAuth users > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36664) in GitLab 13.4. You can automatically link OmniAuth users with existing GitLab users if their email addresses match. -Automatic linking using this method works for all providers -[except the SAML provider](https://gitlab.com/gitlab-org/gitlab/-/issues/338293). For automatic -linking using the SAML provider, see [SAML-specific](saml.md#general-setup) instructions. -As an example, the following configuration is used to enable the auto link -feature for both an **OpenID Connect provider** and a **Twitter OAuth provider**. +The following example enables automatic linking +for the OpenID Connect provider and the Twitter OAuth provider. - **For Omnibus installations** @@ -185,18 +151,23 @@ feature for both an **OpenID Connect provider** and a **Twitter OAuth provider** auto_link_user: ["openid_connect", "twitter"] ``` -## Configure OmniAuth Providers as External +This method of enabling automatic linking works for all providers +[except SAML](https://gitlab.com/gitlab-org/gitlab/-/issues/338293). +To enable automatic linking for SAML, see the [SAML setup instructions](saml.md#general-setup). -You can define which OmniAuth providers you want to be `external`. Users -creating accounts, or logging in by using these `external` providers cannot have -access to internal projects. You must use the full name of the provider, -like `google_oauth2` for Google. Refer to the examples for the full names of the -supported providers. +## Create an external providers list + +You can define a list of external OmniAuth providers. +Users who create accounts or sign in to GitLab through the listed providers do not get access to [internal projects](../public_access/public_access.md#internal-projects-and-groups). + +To define the external providers list, use the full name of the provider, +for example, `google_oauth2` for Google. For provider names, see the +**OmniAuth provider name** column in the [supported providers table](#supported-providers). NOTE: -If you decide to remove an OmniAuth provider from the external providers list, -you must manually update the users that use this method to sign in if you want -their accounts to be upgraded to full internal accounts. +If you remove an OmniAuth provider from the external providers list, +you must manually update the users that use this sign-in method so their +accounts are upgraded to full internal accounts. - **For Omnibus installations** @@ -211,75 +182,70 @@ their accounts to be upgraded to full internal accounts. external_providers: ['twitter', 'google_oauth2'] ``` -## Using Custom OmniAuth Providers +## Use a custom OmniAuth provider NOTE: -The following information only applies for installations from source. +The following information only applies to installations from source. -GitLab uses [OmniAuth](https://github.com/omniauth/omniauth) for authentication and already ships -with a few providers pre-installed, such as LDAP, GitHub, and Twitter. You may also -have to integrate with other authentication solutions. For -these cases, you can use the OmniAuth provider. +If you have to integrate with an authentication solution other than the [OmniAuth](https://github.com/omniauth/omniauth) providers included with GitLab, +you can use a custom OmniAuth provider. -### Steps +These steps are general. Read the OmniAuth provider's documentation for the exact +implementation details. -These steps are fairly general and you must figure out the exact details -from the OmniAuth provider's documentation. +1. Stop GitLab: -- Stop GitLab: - - ```shell - sudo service gitlab stop - ``` + ```shell + sudo service gitlab stop + ``` -- Add the gem to your [`Gemfile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/Gemfile): +1. Add the gem to your [`Gemfile`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/Gemfile): - ```shell - gem "omniauth-your-auth-provider" - ``` + ```shell + gem "omniauth-your-auth-provider" + ``` -- Install the new OmniAuth provider gem by running the following command: +1. Install the new OmniAuth provider gem: - ```shell - sudo -u git -H bundle install --without development test mysql --path vendor/bundle --no-deployment - ``` + ```shell + sudo -u git -H bundle install --without development test mysql --path vendor/bundle --no-deployment + ``` - > These are the same commands you used during initial installation in the [Install Gems section](../install/installation.md#install-gems) with `--path vendor/bundle --no-deployment` instead of `--deployment`. + These commands are the same as the commands for [installing gems](../install/installation.md#install-gems) + during initial installation, with `--path vendor/bundle --no-deployment` instead of `--deployment`. -- Start GitLab: +1. Start GitLab: - ```shell - sudo service gitlab start - ``` + ```shell + sudo service gitlab start + ``` -### Examples +### Custom OmniAuth provider examples -If you have successfully set up a provider that is not shipped with GitLab itself, -please let us know. +If you have successfully set up a provider that is not already integrated with GitLab, +let us know. -While we can't officially support every possible authentication mechanism out there, -we'd like to at least help those with specific needs. +We can't officially support every possible authentication mechanism available, +but we'd like to at least help those with specific needs. -## Enable or disable Sign In with an OmniAuth provider without disabling import sources +## Enable or disable sign-in with an OmniAuth provider without disabling import sources -Administrators are able to enable or disable **Sign In** by using some OmniAuth providers. +Administrators can enable or disable sign-in for some OmniAuth providers. NOTE: -By default, **Sign In** is enabled by using all the OAuth Providers that have been configured in `config/gitlab.yml`. +By default, sign-in is enabled for all the OAuth providers configured in `config/gitlab.yml`. -To enable/disable an OmniAuth provider: +To enable or disable an OmniAuth provider: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, go to **Settings**. -1. Scroll to the **Sign-in Restrictions** section, and click **Expand**. -1. Below **Enabled OAuth Sign-In sources**, select the checkbox for each provider you want to enable or disable. +1. On the left sidebar, select **Settings**. +1. Expand **Sign-in restrictions**. +1. In the **Enabled OAuth authentication sources** section, select or clear the checkbox for each provider you want to enable or disable. -  +## Disable OmniAuth -## Disabling OmniAuth - -Starting from version 11.4 of GitLab, OmniAuth is enabled by default. This only -has an effect if providers are configured and [enabled](#enable-or-disable-sign-in-with-an-omniauth-provider-without-disabling-import-sources). +In GitLab 11.4 and later, OmniAuth is enabled by default. However, OmniAuth only works +if providers are configured and [enabled](#enable-or-disable-sign-in-with-an-omniauth-provider-without-disabling-import-sources). If OmniAuth providers are causing problems even when individually disabled, you can disable the entire OmniAuth subsystem by modifying the configuration file: @@ -299,7 +265,8 @@ can disable the entire OmniAuth subsystem by modifying the configuration file: ## Keep OmniAuth user profiles up to date -You can enable profile syncing from selected OmniAuth providers and for all or for specific user information. +You can enable profile syncing from selected OmniAuth providers. You can sync +all or specific user information. When authenticating using LDAP, the user's name and email are always synced. @@ -318,15 +285,22 @@ When authenticating using LDAP, the user's name and email are always synced. sync_profile_attributes: ['email', 'location'] ``` -## Bypassing two factor authentication +## Bypass two-factor authentication + +> Introduced in GitLab 12.3. -In GitLab 12.3 or later, users can sign in with specified providers _without_ -using two factor authentication. +With certain OmniAuth providers, users can sign in without +using two-factor authentication. -Define the allowed providers using an array (for example, `["twitter", 'google_oauth2']`), -or as `true` or `false` to allow all providers (or none). This option should be -configured only for providers which already have two factor authentication -(default: false). This configuration doesn't apply to SAML. +To bypass two-factor authentication, you can either: + +- Define the allowed providers using an array (for example, `['twitter', 'google_oauth2']`). +- Specify `true` to allow all providers, or `false` to allow none. + +This option should be configured only for providers that already have +two-factor authentication. The default is `false`. + +This configuration doesn't apply to SAML. - **For Omnibus package** @@ -341,14 +315,14 @@ configured only for providers which already have two factor authentication allow_bypass_two_factor: ['twitter', 'google_oauth2'] ``` -## Automatically sign in with provider +## Sign in with a provider automatically You can add the `auto_sign_in_with_provider` setting to your GitLab configuration to redirect login requests to your OmniAuth provider for -authentication. This removes the need to click a button before actually signing in. +authentication. This removes the need to select the provider before signing in. -For example, when using the [Azure v2 integration](azure.md#microsoft-azure-oauth-20-omniauth-provider-v2), set the following to enable auto -sign-in: +For example, to enable automatic sign-in for the +[Azure v2 integration](azure.md#microsoft-azure-oauth-20-omniauth-provider-v2): - **For Omnibus package** @@ -364,10 +338,10 @@ sign-in: ``` Keep in mind that every sign-in attempt is redirected to the OmniAuth -provider; you can't sign in using local credentials. Ensure at least -one of the OmniAuth users has an administrator role. +provider, so you can't sign in using local credentials. Ensure at least +one of the OmniAuth users is an administrator. -You may also bypass the auto sign in feature by browsing to +You can also bypass automatic sign-in by browsing to `https://gitlab.example.com/users/sign_in?auto_sign_in=false`. ## Passwords for users created via OmniAuth @@ -376,11 +350,12 @@ The [Generated passwords for users created through integrated authentication](.. guide provides an overview about how GitLab generates and sets passwords for users created with OmniAuth. -## Custom OmniAuth provider icon +## Use a custom OmniAuth provider icon Most supported providers include a built-in icon for the rendered sign-in button. -After you ensure your image is optimized for rendering at 64 x 64 pixels, -you can override this icon in one of two ways: + +To use your own icon, ensure your image is optimized for rendering at 64 x 64 pixels, +then override the icon in one of two ways: - **Provide a custom image path**: @@ -391,11 +366,11 @@ you can override this icon in one of two ways: to your GitLab configuration file. Read [OpenID Connect OmniAuth provider](../administration/auth/oidc.md) for an example for the OpenID Connect provider. -- **Directly embed an image in a configuration file**: This example creates a Base64-encoded +- **Embed an image directly in a configuration file**: This example creates a Base64-encoded version of your image you can serve through a [Data URL](https://developer.mozilla.org/en-US/docs/Web/HTTP/Basics_of_HTTP/Data_URIs): - 1. Encode your image file with GNU `base64` command (such as `base64 -w 0 <logo.png>`) + 1. Encode your image file with a GNU `base64` command (such as `base64 -w 0 <logo.png>`) which returns a single-line `<base64-data>` string. 1. Add the Base64-encoded data to a custom `icon` parameter in your GitLab configuration file: diff --git a/doc/integration/salesforce.md b/doc/integration/salesforce.md index 9555c762761..68daced3521 100644 --- a/doc/integration/salesforce.md +++ b/doc/integration/salesforce.md @@ -11,7 +11,7 @@ You can integrate your GitLab instance with [Salesforce](https://www.salesforce. ## Create a Salesforce Connected App To enable Salesforce OmniAuth provider, you must use Salesforce's credentials for your GitLab instance. -To get the credentials (a pair of Client ID and Client Secret), you must [create a Connected App](https://help.salesforce.com/articleView?id=connected_app_create.htm&type=5) on Salesforce. +To get the credentials (a pair of Client ID and Client Secret), you must [create a Connected App](https://help.salesforce.com/s/articleView?id=connected_app_create.htm&type=5) on Salesforce. 1. Sign in to [Salesforce](https://login.salesforce.com/). @@ -48,7 +48,7 @@ To get the credentials (a pair of Client ID and Client Secret), you must [create sudo -u git -H editor config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration: @@ -79,7 +79,9 @@ To get the credentials (a pair of Client ID and Client Secret), you must [create  1. Save the configuration file. -1. [Reconfigure GitLab]( ../administration/restart_gitlab.md#omnibus-gitlab-reconfigure ) or [restart GitLab]( ../administration/restart_gitlab.md#installations-from-source ) for the changes to take effect if you installed GitLab via Omnibus or from source respectively. +1. [Reconfigure GitLab](../administration/restart_gitlab.md#omnibus-gitlab-reconfigure) or + [restart GitLab](../administration/restart_gitlab.md#installations-from-source) for the changes + to take effect if you installed GitLab via Omnibus or from source respectively. On the sign in page, there should now be a Salesforce icon below the regular sign in form. Click the icon to begin the authentication process. Salesforce asks the user to sign in and authorize the GitLab application. diff --git a/doc/integration/saml.md b/doc/integration/saml.md index 876eb7ba80b..47a35cf21a8 100644 --- a/doc/integration/saml.md +++ b/doc/integration/saml.md @@ -53,7 +53,7 @@ in your SAML IdP: sudo -u git -H editor config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. 1. To allow your users to use SAML to sign up without having to manually create an account first, add the following values to your configuration: @@ -269,7 +269,7 @@ Example: idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', idp_sso_target_url: 'https://login.example.com/idp', issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' } } ``` @@ -311,7 +311,7 @@ The requirements are the same as the previous settings: idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', idp_sso_target_url: 'https://login.example.com/idp', issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' } } ``` @@ -335,7 +335,7 @@ The requirements are the same as the previous settings: idp_cert_fingerprint: '43:51:43:a1:b5:fc:8b:b7:0a:3a:a9:b1:0f:66:73:a8', idp_sso_target_url: 'https://login.example.com/idp', issuer: 'https://gitlab.example.com', - name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:transient' + name_identifier_format: 'urn:oasis:names:tc:SAML:2.0:nameid-format:persistent' } } ``` diff --git a/doc/integration/security_partners/index.md b/doc/integration/security_partners/index.md index b8c7a0163f5..2c7641124a0 100644 --- a/doc/integration/security_partners/index.md +++ b/doc/integration/security_partners/index.md @@ -18,7 +18,7 @@ each security partner: - [Checkmarx](https://checkmarx.atlassian.net/wiki/spaces/SD/pages/1929937052/GitLab+Integration) - [Deepfactor](https://docs.deepfactor.io/hc/en-us/articles/1500008981941) - [GrammaTech](https://www.grammatech.com/codesonar-gitlab-integration) -- [Indeni](https://indeni.com/doc-indeni-cloudrail/integrate-with-ci-cd/gitlab-instructions/) +- [Indeni](https://cloudrail.app/doc/integrate-with-ci-cd/gitlab-instructions/) - [JScrambler](https://docs.jscrambler.com/code-integrity/documentation/gitlab-ci-integration) - [Semgrep](https://semgrep.dev/for/gitlab) - [StackHawk](https://docs.stackhawk.com/continuous-integration/gitlab.html) diff --git a/doc/integration/twitter.md b/doc/integration/twitter.md index f8dd552ec6a..50ef04681f0 100644 --- a/doc/integration/twitter.md +++ b/doc/integration/twitter.md @@ -53,7 +53,7 @@ Twitter. Twitter generates a client ID and secret key for you to use. sudo -u git -H editor config/gitlab.yml ``` -1. See [Initial OmniAuth Configuration](omniauth.md#initial-omniauth-configuration) for initial settings. +1. See [Configure initial settings](omniauth.md#configure-initial-settings) for initial settings. 1. Add the provider configuration: diff --git a/doc/operations/error_tracking.md b/doc/operations/error_tracking.md index 88785196f6e..70e3115a98e 100644 --- a/doc/operations/error_tracking.md +++ b/doc/operations/error_tracking.md @@ -169,5 +169,6 @@ You can do so by managing client keys with the [error tracking API](../api/error #### Limitations -The Integrated Error Tracking feature was built and tested with Sentry SDK for Ruby. Other languages and frameworks -are not tested and might not work. Check [the compatibility issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340178) for more information. +The Integrated Error Tracking feature was built and tested with Sentry SDK for Ruby on Rails. +Support for other languages and frameworks is not guaranteed. For up-to-date information, see the +[compatibility issue](https://gitlab.com/gitlab-org/gitlab/-/issues/340178). diff --git a/doc/operations/feature_flags.md b/doc/operations/feature_flags.md index 2af4ee47292..2ef193b0f5d 100644 --- a/doc/operations/feature_flags.md +++ b/doc/operations/feature_flags.md @@ -38,11 +38,12 @@ with GitLab, so it's up to developers to use a compatible client library and To create and enable a feature flag: -1. Navigate to your project's **Deployments > Feature Flags**. -1. Click the **New feature flag** button. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Feature Flags**. +1. Select **New feature flag**. 1. Enter a name that starts with a letter and contains only lowercase letters, digits, underscores (`_`), or dashes (`-`), and does not end with a dash (`-`) or underscore (`_`). -1. Enter a description (optional, 255 characters max). +1. Optional. Enter a description (255 characters maximum). 1. Enter details about how the flag should be applied: - In GitLab 13.0 and earlier, add **Environment specs**. For each environment, include the **Status** (default enabled) and [**Rollout strategy**](#rollout-strategy-legacy) @@ -50,9 +51,9 @@ To create and enable a feature flag: - In GitLab 13.1 and later, add Feature Flag [**Strategies**](#feature-flag-strategies). For each strategy, include the **Type** (defaults to [**All users**](#all-users)) and **Environments** (defaults to all environments). -1. Click **Create feature flag**. +1. Select **Create feature flag**. -You can change these settings by clicking the **{pencil}** (edit) button +To change these settings, select **Edit** (**{pencil}**). next to any feature flag in the list. ## Maximum number of feature flags @@ -91,7 +92,7 @@ and the supported strategies are: Strategies can be added to feature flags when [creating a feature flag](#create-a-feature-flag), or by editing an existing feature flag after creation by navigating to **Deployments > Feature Flags** -and clicking **{pencil}** (edit). +and selecting **Edit** (**{pencil}**). ### All users @@ -204,14 +205,15 @@ For example: To create a user list: -1. In your project, navigate to **Deployments > Feature Flags**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Feature Flags**. 1. Select **View user lists** 1. Select **New user list**. 1. Enter a name for the list. 1. Select **Create**. -You can view a list's User IDs by clicking the **{pencil}** (edit) button next to it. -When viewing a list, you can rename it by clicking the **Edit** button. +You can view a list's User IDs by selecting **Edit** (**{pencil}**) next to it. +When viewing a list, you can rename it by selecting **Edit** (**{pencil}**). #### Add users to a user list @@ -219,12 +221,13 @@ When viewing a list, you can rename it by clicking the **Edit** button. To add users to a user list: -1. In your project, navigate to **Deployments > Feature Flags**. -1. Click on the **{pencil}** (edit) button next to the list you want to add users to. -1. Click on **Add Users**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Feature Flags**. +1. Select **Edit** (**{pencil}**) next to the list you want to add users to. +1. Select **Add Users**. 1. Enter the user IDs as a comma-separated list of values. For example, `user@example.com, user2@example.com`, or `username1,username2,username3`, and so on. -1. Click on **Add**. +1. Select **Add**. #### Remove users from a user list @@ -232,9 +235,10 @@ To add users to a user list: To remove users from a user list: -1. In your project, navigate to **Deployments > Feature Flags**. -1. Click on the **{pencil}** (edit) button next to the list you want to change. -1. Click on the **{remove}** (remove) button next to the ID you want to remove. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Feature Flags**. +1. Select **Edit** (**{pencil}**) next to the list you want to change. +1. Select **Remove** (**{remove}**) next to the ID you want to remove. ## Rollout strategy (legacy) @@ -270,21 +274,23 @@ See [this video tutorial](https://www.youtube.com/watch?v=CAJY2IGep7Y) for help In [GitLab 13.0 and earlier](https://gitlab.com/gitlab-org/gitlab/-/issues/8621), to disable a feature flag for a specific environment: -1. Navigate to your project's **Deployments > Feature Flags**. -1. For the feature flag you want to disable, click the Pencil icon. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Feature Flags**. +1. For the feature flag you want to disable, select **Edit** (**{pencil}**). 1. To disable the flag: - In GitLab 13.0 and earlier: Slide the Status toggle for the environment. Or, to delete the - environment spec, on the right, click the **Remove (X)** icon. + environment spec, on the right, select **Remove (X)**. - In GitLab 13.1 and later: For each strategy it applies to, under **Environments**, delete the environment. -1. Click **Save changes**. +1. Select **Save changes**. ## Disable a feature flag for all environments To disable a feature flag for all environments: -1. Navigate to your project's **Deployments > Feature Flags**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Feature Flags**. 1. For the feature flag you want to disable, slide the Status toggle to **Disabled**. The feature flag is displayed on the **Disabled** tab. @@ -298,8 +304,9 @@ Then prepare your application with a client library. To get the access credentials that your application needs to communicate with GitLab: -1. Navigate to your project's **Deployments > Feature Flags**. -1. Click the **Configure** button to view the following: +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Deployments > Feature Flags**. +1. Select **Configure** to view the following: - **API URL**: URL where the client (application) connects to get a list of feature flags. - **Instance ID**: Unique token that authorizes the retrieval of the feature flags. - **Application name**: The name of the *environment* the application runs in @@ -403,6 +410,31 @@ else end ``` +### Unleash Proxy example + +As of [Unleash Proxy](https://docs.getunleash.io/sdks/unleash-proxy) version +0.2, the proxy is compatible with feature flags. To run a Docker container to +connect to your project's feature flags, run the following command: + +```shell +docker run \ + -e UNLEASH_PROXY_SECRETS=<secret> \ + -e UNLEASH_URL=<project feature flags URL> \ + -e UNLEASH_INSTANCE_ID=<project feature flags instance ID> \ + -e UNLEASH_APP_NAME=<project environment> \ + -e UNLEASH_API_TOKEN=<tokenNotUsed> \ + -p 3000:3000 \ + unleashorg/unleash-proxy +``` + +| Variable | Value | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `UNLEASH_PROXY_SECRETS` | Shared secret used to configure an [Unleash Proxy client](https://docs.getunleash.io/sdks/unleash-proxy#how-to-connect-to-the-proxy). | +| `UNLEASH_URL` | Your project's API URL. For more details, read [Get access credentials](#get-access-credentials). | +| `UNLEASH_INSTANCE_ID` | Your project's Instance ID. For more details, read [Get access credentials](#get-access-credentials). | +| `UNLEASH_APP_NAME` | The name of the environment the application runs in. For more details, read [Get access credentials](#get-access-credentials). | +| `UNLEASH_API_TOKEN` | Required to start the Unleash Proxy, but not used to connect to GitLab. Can be set to any value. | + ## Feature Flag Related Issues **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/36617) in GitLab 13.2. diff --git a/doc/operations/incident_management/incidents.md b/doc/operations/incident_management/incidents.md index 2d9ef21f1ce..64dea795d3c 100644 --- a/doc/operations/incident_management/incidents.md +++ b/doc/operations/incident_management/incidents.md @@ -19,20 +19,21 @@ You can create an incident manually or automatically. ### Create incidents manually -If you have at least Guest [permissions](../../user/permissions.md), to create an -Incident, you have two options to do this manually. +> - [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3. +> - [Permission changed](https://gitlab.com/gitlab-org/gitlab/-/issues/336624) from Guest to Reporter in GitLab 14.5. -**From the Incidents List:** +If you have at least Reporter [permissions](../../user/permissions.md), +you can create an incident manually from the Incidents List or the Issues List. -> [Moved](https://gitlab.com/gitlab-org/monitor/monitor/-/issues/24) to GitLab Free in 13.3. +To create an incident from the Incidents List: -- Navigate to **Monitor > Incidents** and click **Create Incident**. -- Create a new issue using the `incident` template available when creating it. -- Create a new issue and assign the `incident` label to it. +1. Navigate to **Monitor > Incidents** and click **Create Incident**. +1. Create a new issue using the `incident` template available when creating it. +1. Create a new issue and assign the `incident` label to it.  -**From the Issues List:** +To create an incident from the Issues List: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/230857) in GitLab 13.4. diff --git a/doc/operations/incident_management/integrations.md b/doc/operations/incident_management/integrations.md index 1426148d163..92f5a50b1c3 100644 --- a/doc/operations/incident_management/integrations.md +++ b/doc/operations/incident_management/integrations.md @@ -196,6 +196,27 @@ WARNING: Using your authorization key in the URL is insecure, as it's visible in server logs. We recommend using one of the above header options if your tooling supports it. +## Response body + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/342730) in GitLab 14.5. + +The JSON response body contains a list of any alerts created within the request: + +```json +[ + { + "iid": 1, + "title": "Incident title" + }, + { + "iid": 2, + "title": "Second Incident title" + } +] +``` + +Successful responses return a `200` response code. + ## Triggering test alerts > [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3066) in GitLab in 13.2. diff --git a/doc/push_rules/push_rules.md b/doc/push_rules/push_rules.md index 3d0e4fdbe49..37aeb335825 100644 --- a/doc/push_rules/push_rules.md +++ b/doc/push_rules/push_rules.md @@ -5,7 +5,7 @@ info: "To determine the technical writer assigned to the Stage/Group associated type: reference, howto --- -# Push Rules **(PREMIUM)** +# Push rules **(PREMIUM)** Gain additional control over what can and can't be pushed to your repository by using regular expressions to reject pushes based on commit contents, branch names or file details. diff --git a/doc/raketasks/backup_restore.md b/doc/raketasks/backup_restore.md index 89b727cf570..cd541e7827f 100644 --- a/doc/raketasks/backup_restore.md +++ b/doc/raketasks/backup_restore.md @@ -22,7 +22,7 @@ backups with your object storage provider, if desired. ## Requirements -To be able to backup and restore, ensure that Rsync is installed on your +To be able to back up and restore, ensure that Rsync is installed on your system. If you installed GitLab: - _Using the Omnibus package_, you're all set. @@ -61,7 +61,7 @@ including: - Container Registry images - GitLab Pages content - Snippets -- [Group wikis](../user/project/wiki/index.md#group-wikis) +- [Group wikis](../user/project/wiki/group.md) Backups do not include: @@ -74,7 +74,7 @@ GitLab does not back up any configuration files, SSL certificates, or system files. You are highly advised to read about [storing configuration files](#storing-configuration-files). WARNING: -The backup command requires [additional parameters](#backup-and-restore-for-installations-using-pgbouncer) when +The backup command requires [additional parameters](#back-up-and-restore-for-installations-using-pgbouncer) when your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster. Depending on your version of GitLab, use the following command if you installed @@ -168,7 +168,7 @@ as its key defeats the purpose of using encryption in the first place. WARNING: The secrets file is essential to preserve your database encryption key. -At the very **minimum**, you must backup: +At the very **minimum**, you must back up: For Omnibus: @@ -187,7 +187,7 @@ the GitLab container according to the documentation, it should be in the For [GitLab Helm chart installations](https://gitlab.com/gitlab-org/charts/gitlab) on a Kubernetes cluster, you must follow the -[Backup the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#backup-the-secrets) +[Back up the secrets](https://docs.gitlab.com/charts/backup-restore/backup.html#backup-the-secrets) instructions. You may also want to back up any TLS keys and certificates, and your @@ -409,6 +409,8 @@ For Omnibus GitLab packages: ##### S3 Encrypted Buckets +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/64765) in GitLab 14.3. + AWS supports these [modes for server side encryption](https://docs.aws.amazon.com/AmazonS3/latest/userguide/serv-side-encryption.html): - Amazon S3-Managed Keys (SSE-S3) @@ -787,7 +789,7 @@ For installations from source: #### Configuring cron to make daily backups WARNING: -The following cron jobs do not [backup your GitLab configuration files](#storing-configuration-files) +The following cron jobs do not [back up your GitLab configuration files](#storing-configuration-files) or [SSH host keys](https://superuser.com/questions/532040/copy-ssh-keys-from-one-server-to-another-server/532079#532079). You can schedule a cron job that backs up your repositories and GitLab metadata. @@ -974,7 +976,7 @@ message. Install the [correct GitLab version](https://packages.gitlab.com/gitlab and then try again. WARNING: -The restore command requires [additional parameters](#backup-and-restore-for-installations-using-pgbouncer) when +The restore command requires [additional parameters](#back-up-and-restore-for-installations-using-pgbouncer) when your installation is using PgBouncer, for either performance reasons or when using it with a Patroni cluster. Next, restore `/etc/gitlab/gitlab-secrets.json` if necessary, @@ -1020,7 +1022,7 @@ docker exec -it <name of container> gitlab-ctl stop sidekiq # Verify that the processes are all down before continuing docker exec -it <name of container> gitlab-ctl status -# Run the restore +# Run the restore. NOTE: "_gitlab_backup.tar" is omitted from the name docker exec -it <name of container> gitlab-backup restore BACKUP=11493107454_2018_04_25_10.6.4-ce # Restart the GitLab container @@ -1215,9 +1217,9 @@ There are a few possible downsides to this: There is an **experimental** script that attempts to automate this process in [the Geo team Runbooks project](https://gitlab.com/gitlab-org/geo-team/runbooks/-/tree/main/experimental-online-backup-through-rsync). -## Backup and restore for installations using PgBouncer +## Back up and restore for installations using PgBouncer -Do NOT backup or restore GitLab through a PgBouncer connection. These +Do NOT back up or restore GitLab through a PgBouncer connection. These tasks must [bypass PgBouncer and connect directly to the PostgreSQL primary database node](#bypassing-pgbouncer), or they cause a GitLab outage. @@ -1377,18 +1379,30 @@ after which users must reactivate 2FA. 1. Enter the database console: - For Omnibus GitLab packages: + For Omnibus GitLab 14.1 and earlier: ```shell sudo gitlab-rails dbconsole ``` - For installations from source: + For Omnibus GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For installations from source, GitLab 14.1 and earlier: ```shell sudo -u git -H bundle exec rails dbconsole -e production ``` + For installations from source, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + 1. Examine the `ci_group_variables` and `ci_variables` tables: ```sql @@ -1411,18 +1425,30 @@ You may need to reconfigure or restart GitLab for the changes to take effect. 1. Enter the database console: - For Omnibus GitLab packages: + For Omnibus GitLab 14.1 and earlier: ```shell sudo gitlab-rails dbconsole ``` - For installations from source: + For Omnibus GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For installations from source, GitLab 14.1 and earlier: ```shell sudo -u git -H bundle exec rails dbconsole -e production ``` + For installations from source, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + 1. Clear all tokens for projects, groups, and the entire instance: WARNING: @@ -1448,18 +1474,30 @@ You may need to reconfigure or restart GitLab for the changes to take effect. 1. Enter the database console: - For Omnibus GitLab packages: + For Omnibus GitLab 14.1 and earlier: ```shell sudo gitlab-rails dbconsole ``` - For installations from source: + For Omnibus GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For installations from source, GitLab 14.1 and earlier: ```shell sudo -u git -H bundle exec rails dbconsole -e production ``` + For installations from source, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + 1. Clear all the tokens for pending jobs: ```sql @@ -1480,18 +1518,30 @@ The fix is to truncate the `web_hooks` table: 1. Enter the database console: - For Omnibus GitLab packages: + For Omnibus GitLab 14.1 and earlier: ```shell sudo gitlab-rails dbconsole ``` - For installations from source: + For Omnibus GitLab 14.2 and later: + + ```shell + sudo gitlab-rails dbconsole --database main + ``` + + For installations from source, GitLab 14.1 and earlier: ```shell sudo -u git -H bundle exec rails dbconsole -e production ``` + For installations from source, GitLab 14.2 and later: + + ```shell + sudo -u git -H bundle exec rails dbconsole -e production --database main + ``` + 1. Truncate the table: ```sql diff --git a/doc/raketasks/import.md b/doc/raketasks/import.md index 7f817f9c422..45d51fe9dfa 100644 --- a/doc/raketasks/import.md +++ b/doc/raketasks/import.md @@ -58,7 +58,7 @@ To import bare repositories into a GitLab instance: - Omnibus Installation ```shell - sudo gitlab-rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-$(date "+%Y-%m-%d")'] + sudo gitlab-rake gitlab:import:repos["/var/opt/gitlab/git-data/repository-import-$(date "+%Y-%m-%d")"] ``` - Installation from source. Before running this command you need to change to the directory where @@ -66,7 +66,7 @@ To import bare repositories into a GitLab instance: ```shell cd /home/git/gitlab - sudo -u git -H bundle exec rake gitlab:import:repos['/var/opt/gitlab/git-data/repository-import-$(date "+%Y-%m-%d")'] RAILS_ENV=production + sudo -u git -H bundle exec rake gitlab:import:repos["/var/opt/gitlab/git-data/repository-import-$(date "+%Y-%m-%d")"] RAILS_ENV=production ``` ## Example output diff --git a/doc/raketasks/spdx.md b/doc/raketasks/spdx.md index f330698daba..18f058f695e 100644 --- a/doc/raketasks/spdx.md +++ b/doc/raketasks/spdx.md @@ -4,7 +4,7 @@ group: Composition Analysis info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# SPDX license list import **(PREMIUM SELF)** +# SPDX license list import **(ULTIMATE SELF)** GitLab provides a Rake task for uploading a fresh copy of the [SPDX license list](https://spdx.org/licenses/) to a GitLab instance. This list is needed for matching the names of [License Compliance policies](../user/compliance/license_compliance/index.md). diff --git a/doc/security/rack_attack.md b/doc/security/rack_attack.md index b0bebc5a956..a8b55007d2e 100644 --- a/doc/security/rack_attack.md +++ b/doc/security/rack_attack.md @@ -1,195 +1,9 @@ --- -stage: Manage -group: Access -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, howto +redirect_to: '../user/admin_area/settings/protected_paths.md' +remove_date: '2022-01-14' --- -# Rack Attack initializer **(FREE SELF)** +This document was moved to [another location](../user/admin_area/settings/protected_paths.md). -[Rack Attack](https://github.com/kickstarter/rack-attack), also known as Rack::Attack, is a Ruby gem -that is meant to protect GitLab with the ability to customize throttling and -to block user IP addresses. - -You can prevent brute-force passwords attacks, scrapers, or any other offenders -by throttling requests from IP addresses that are making large volumes of requests. -If you find throttling is not enough to protect you against abusive clients, -Rack Attack offers IP whitelisting, blacklisting, Fail2ban style filtering, and -tracking. - -For more information on how to use these options see the [Rack Attack README](https://github.com/kickstarter/rack-attack/blob/master/README.md). - -NOTE: -See -[User and IP rate limits](../user/admin_area/settings/user_and_ip_rate_limits.md) -for simpler limits that are configured in the UI. - -NOTE: -Starting with GitLab 11.2, Rack Attack is disabled by default. If your -instance is not exposed to the public internet, it is recommended that you leave -Rack Attack disabled. - -## Behavior - -If set up as described in the [Settings](#settings) section below, two behaviors -are enabled: - -- Protected paths are throttled. -- Failed authentications for Git and container registry requests trigger a temporary IP ban. - -### Protected paths throttle - -GitLab responds with HTTP status code `429` to POST requests at protected paths -that exceed 10 requests per minute per IP address. - -By default, protected paths are: - -- `/users/password` -- `/users/sign_in` -- `/api/#{API::API.version}/session.json` -- `/api/#{API::API.version}/session` -- `/users` -- `/users/confirmation` -- `/unsubscribes/` -- `/import/github/personal_access_token` -- `/admin/session` - -See [User and IP rate limits](../user/admin_area/settings/user_and_ip_rate_limits.md#response-headers) for the headers responded to blocked requests. - -For example, the following are limited to a maximum 10 requests per minute: - -- User sign-in -- User sign-up (if enabled) -- User password reset - -After 10 requests, the client must wait a minute before it can -try again. - -### Git and container registry failed authentication ban - -GitLab responds with HTTP status code `403` for 1 hour, if 30 failed -authentication requests were received in a 3-minute period from a single IP address. - -This applies only to Git requests and container registry (`/jwt/auth`) requests -(combined). - -This limit: - -- Is reset by requests that authenticate successfully. For example, 29 - failed authentication requests followed by 1 successful request, followed by 29 - more failed authentication requests would not trigger a ban. -- Does not apply to JWT requests authenticated by `gitlab-ci-token`. - -No response headers are provided. - -## Settings - -**Omnibus GitLab** - -1. Open `/etc/gitlab/gitlab.rb` with your editor -1. Add the following: - - ```ruby - gitlab_rails['rack_attack_git_basic_auth'] = { - 'enabled' => true, - 'ip_whitelist' => ["127.0.0.1"], - 'maxretry' => 10, # Limit the number of Git HTTP authentication attempts per IP - 'findtime' => 60, # Reset the auth attempt counter per IP after 60 seconds - 'bantime' => 3600 # Ban an IP for one hour (3600s) after too many auth attempts - } - ``` - -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -The following settings can be configured: - -- `enabled`: By default this is set to `false`. Set this to `true` to enable Rack Attack. -- `ip_whitelist`: Whitelist any IPs from being blocked. They must be formatted as strings within a Ruby array. - CIDR notation is supported in GitLab 12.1 and later. - For example, `["127.0.0.1", "127.0.0.2", "127.0.0.3", "192.168.0.1/24"]`. -- `maxretry`: The maximum amount of times a request can be made in the - specified time. -- `findtime`: The maximum amount of time that failed requests can count against an IP - before it's blacklisted (in seconds). -- `bantime`: The total amount of time that a blacklisted IP is blocked (in - seconds). - -**Installations from source** - -These settings can be found in `config/initializers/rack_attack.rb`. If you are -missing `config/initializers/rack_attack.rb`, the following steps need to be -taken in order to enable protection for your GitLab instance: - -1. In `config/application.rb` find and uncomment the following line: - - ```ruby - config.middleware.use Rack::Attack - ``` - -1. Restart GitLab: - - ```shell - sudo service gitlab restart - ``` - -If you want more restrictive/relaxed throttle rules, edit -`config/initializers/rack_attack.rb` and change the `limit` or `period` values. -For example, you can set more relaxed throttle rules with -`limit: 3` and `period: 1.seconds`, allowing 3 requests per second. -You can also add other paths to the protected list by adding to `paths_to_be_protected` -variable. If you change any of these settings you must restart your -GitLab instance. - -## Remove blocked IPs from Rack Attack via Redis - -In case you want to remove a blocked IP, follow these steps: - -1. Find the IPs that have been blocked in the production log: - - ```shell - grep "Rack_Attack" /var/log/gitlab/gitlab-rails/auth.log - ``` - -1. Since the blacklist is stored in Redis, you need to open up `redis-cli`: - - ```shell - /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket - ``` - -1. You can remove the block using the following syntax, replacing `<ip>` with - the actual IP that is blacklisted: - - ```plaintext - del cache:gitlab:rack::attack:allow2ban:ban:<ip> - ``` - -1. Confirm that the key with the IP no longer shows up: - - ```plaintext - keys *rack::attack* - ``` - -1. Optionally, add the IP to the whitelist to prevent it from being blacklisted - again (see [settings](#settings)). - -## Troubleshooting - -### Rack attack is blacklisting the load balancer - -Rack Attack may block your load balancer if all traffic appears to come from -the load balancer. In that case, you must: - -1. [Configure `nginx[real_ip_trusted_addresses]`](https://docs.gitlab.com/omnibus/settings/nginx.html#configuring-gitlab-trusted_proxies-and-the-nginx-real_ip-module). - This keeps users' IPs from being listed as the load balancer IPs. -1. Whitelist the load balancer's IP address(es) in the Rack Attack [settings](#settings). -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -1. [Remove the block via Redis.](#remove-blocked-ips-from-rack-attack-via-redis) +<!-- This redirect file can be deleted after <2022-01-14>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/security/rate_limits.md b/doc/security/rate_limits.md index 4585748ffc2..9d49297c9de 100644 --- a/doc/security/rate_limits.md +++ b/doc/security/rate_limits.md @@ -14,18 +14,22 @@ For GitLab.com, please see Rate limiting is a common technique used to improve the security and durability of a web application. -For example, a simple script can make thousands of web requests per second. -Whether malicious, apathetic, or just a bug, your application and infrastructure -may not be able to cope with the load. For more details, see +For example, a simple script can make thousands of web requests per second. The requests could be: + +- Malicious. +- Apathetic. +- Just a bug. + +Your application and infrastructure may not be able to cope with the load. For more details, see [Denial-of-service attack](https://en.wikipedia.org/wiki/Denial-of-service_attack). Most cases can be mitigated by limiting the rate of requests from a single IP address. Most [brute-force attacks](https://en.wikipedia.org/wiki/Brute-force_attack) are similarly mitigated by a rate limit. -## Admin Area settings +## Configurable limits -These are rate limits you can set in the Admin Area of your instance: +You can set these rate limits in the Admin Area of your instance: - [Import/Export rate limits](../user/admin_area/settings/import_export_rate_limits.md) - [Issues rate limits](../user/admin_area/settings/rate_limit_on_issues_creation.md) @@ -38,14 +42,40 @@ These are rate limits you can set in the Admin Area of your instance: - [Files API rate limits](../user/admin_area/settings/files_api_rate_limits.md) - [Deprecated API rate limits](../user/admin_area/settings/deprecated_api_rate_limits.md) +You can set these rate limits using the Rails console: + +- [Webhook rate limit](../administration/instance_limits.md#webhook-rate-limit) + +## Failed authentication ban for Git and container registry + +GitLab returns HTTP status code `403` for 1 hour, if 30 failed authentication requests were received +in a 3-minute period from a single IP address. This applies only to combined: + +- Git requests. +- Container registry (`/jwt/auth`) requests. + +This limit: + +- Is reset by requests that authenticate successfully. For example, 29 failed authentication + requests followed by 1 successful request, followed by 29 more failed authentication requests + would not trigger a ban. +- Does not apply to JWT requests authenticated by `gitlab-ci-token`. +- Is disabled by default. + +No response headers are provided. + +For configuration information, see +[Omnibus GitLab configuration options](https://docs.gitlab.com/omnibus/settings/configuration.html#configure-a-failed-authentication-ban). + ## Non-configurable limits ### Repository archives > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25750) in GitLab 12.9. -There is a rate limit for [downloading repository archives](../api/repositories.md#get-file-archive), -which applies to the project and to the user initiating the download either through the UI or the API. +A rate limit for [downloading repository archives](../api/repositories.md#get-file-archive) is +available. The limit applies to the project and to the user initiating the download either through +the UI or the API. The **rate limit** is 5 requests per minute per user. @@ -57,8 +87,52 @@ There is a rate limit for [testing webhooks](../user/project/integrations/webhoo The **rate limit** is 5 requests per minute per user. -## Rack Attack initializer +## Troubleshooting + +### Rack Attack is denylisting the load balancer + +Rack Attack may block your load balancer if all traffic appears to come from +the load balancer. In that case, you must: + +1. [Configure `nginx[real_ip_trusted_addresses]`](https://docs.gitlab.com/omnibus/settings/nginx.html#configuring-gitlab-trusted_proxies-and-the-nginx-real_ip-module). + This keeps users' IPs from being listed as the load balancer IPs. +1. Allowlist the load balancer's IP addresses. +1. Reconfigure GitLab: + + ```shell + sudo gitlab-ctl reconfigure + ``` + +### Remove blocked IPs from Rack Attack with Redis + +To remove a blocked IP: + +1. Find the IPs that have been blocked in the production log: + + ```shell + grep "Rack_Attack" /var/log/gitlab/gitlab-rails/auth.log + ``` + +1. Since the denylist is stored in Redis, you must open up `redis-cli`: + + ```shell + /opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket + ``` + +1. You can remove the block using the following syntax, replacing `<ip>` with + the actual IP that is denylisted: + + ```plaintext + del cache:gitlab:rack::attack:allow2ban:ban:<ip> + ``` + +1. Confirm that the key with the IP no longer shows up: + + ```plaintext + keys *rack::attack* + ``` + + By default, the [`keys` command is disabled](https://docs.gitlab.com/omnibus/settings/redis.html#renamed-commands). -This method of rate limiting is cumbersome, but has some advantages. It allows -throttling of specific paths, and is also integrated into Git and container -registry requests. See [Rack Attack initializer](rack_attack.md). +1. Optionally, add [the IP to the allowlist](https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-rack-attack) + to prevent it being denylisted again. diff --git a/doc/security/token_overview.md b/doc/security/token_overview.md index 2a971b21840..333548fa1c9 100644 --- a/doc/security/token_overview.md +++ b/doc/security/token_overview.md @@ -117,4 +117,9 @@ Instead, API calls can be passed an access token using headers, like [the `Priva Tokens can also be stored using a [Git credential storage](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage). +Tokens should not be committed to your source code. Instead, consider an approach such as [using external secrets in CI](../ci/secrets/index.md). + When creating a scoped token, consider using the most limited scope possible to reduce the impact of accidentally leaking the token. + +When creating a token, consider setting a token that expires when your task is complete. For example, if performing a one-off import, set the +token to expire after a few hours or a day. This reduces the impact of a token that is accidentally leaked because it is useless when it expires. diff --git a/doc/subscriptions/bronze_starter.md b/doc/subscriptions/bronze_starter.md index bb2a2303d29..78a9e324ada 100644 --- a/doc/subscriptions/bronze_starter.md +++ b/doc/subscriptions/bronze_starter.md @@ -50,15 +50,15 @@ the tiers are no longer mentioned in GitLab documentation: - [Group management through LDAP](../administration/auth/ldap/ldap-troubleshooting.md#group-memberships) - Syncing information through LDAP: - Groups: [one group](../administration/auth/ldap/ldap-troubleshooting.md#sync-one-group), - [all groups programmatically](../administration/auth/ldap/index.md#group-sync), - [group sync schedule](../administration/auth/ldap/index.md#adjust-ldap-group-sync-schedule), and + [all groups programmatically](../administration/auth/ldap/ldap_synchronization.md#group-sync), + [group sync schedule](../administration/auth/ldap/ldap_synchronization.md#adjust-ldap-group-sync-schedule), and [all groups manually](../administration/auth/ldap/ldap-troubleshooting.md#sync-all-groups) - [Configuration settings](../administration/auth/ldap/index.md#ldap-sync-configuration-settings) - - Users: [all users](../administration/auth/ldap/index.md#user-sync), - [administrators](../administration/auth/ldap/index.md#administrator-sync), - [user sync schedule](../administration/auth/ldap/index.md#adjust-ldap-user-sync-schedule) - - [Adding group links](../administration/auth/ldap/index.md#add-group-links) - - [Lock memberships to LDAP synchronization](../administration/auth/ldap/index.md#global-group-memberships-lock) + - Users: [all users](../administration/auth/ldap/ldap_synchronization.md#user-sync), + [administrators](../administration/auth/ldap/ldap_synchronization.md#administrator-sync), + [user sync schedule](../administration/auth/ldap/ldap_synchronization.md#adjust-ldap-user-sync-schedule) + - [Adding group links](../administration/auth/ldap/ldap_synchronization.md#add-group-links) + - [Lock memberships to LDAP synchronization](../administration/auth/ldap/ldap_synchronization.md#global-group-memberships-lock) - Rake tasks for [LDAP tasks](../administration/raketasks/ldap.md), including [syncing groups](../administration/raketasks/ldap.md#run-a-group-sync) - Logging: @@ -92,11 +92,11 @@ the tiers are no longer mentioned in GitLab documentation: - [Pull mirroring](../user/project/repository/mirror/pull.md) outside repositories in a GitLab repository - [Overwrite diverged branches](../user/project/repository/mirror/pull.md#overwrite-diverged-branches) - [Trigger pipelines for mirror updates](../user/project/repository/mirror/pull.md#trigger-pipelines-for-mirror-updates) - - [Hard failures](../user/project/repository/mirror/pull.md#hard-failure) when mirroring fails + - [Fix hard failures when mirroring](../user/project/repository/mirror/pull.md#fix-hard-failures-when-mirroring) - [Trigger pull mirroring from the API](../user/project/repository/mirror/pull.md#trigger-an-update-by-using-the-api) - [Mirror only protected branches](../user/project/repository/mirror/index.md#mirror-only-protected-branches) - [Bidirectional mirroring](../user/project/repository/mirror/bidirectional.md) - - [Mirror with Perforce Helix via Git Fusion](../user/project/repository/mirror/bidirectional.md#mirror-with-perforce-helix-via-git-fusion) + - [Mirror with Perforce Helix with Git Fusion](../user/project/repository/mirror/bidirectional.md#mirror-with-perforce-helix-with-git-fusion) - Runners: - Run pipelines in the parent project [for merge requests from a forked project](../ci/pipelines/merge_request_pipelines.md#run-pipelines-in-the-parent-project-for-merge-requests-from-a-forked-project) - [Shared runners pipeline minutes quota](../user/admin_area/settings/continuous_integration.md#shared-runners-pipeline-minutes-quota) diff --git a/doc/subscriptions/gitlab_com/index.md b/doc/subscriptions/gitlab_com/index.md index a8e02251b64..a26feb6d97e 100644 --- a/doc/subscriptions/gitlab_com/index.md +++ b/doc/subscriptions/gitlab_com/index.md @@ -311,8 +311,8 @@ Your own runners can still be used even if you reach your limits. If you're using GitLab SaaS, you can purchase additional CI minutes so your pipelines aren't blocked after you have used all your CI minutes from your -main quota. You can find pricing for additional CI/CD minutes in the -[GitLab Customers Portal](https://customers.gitlab.com/plans). Additional minutes: +main quota. You can find pricing for additional CI/CD minutes on the +[GitLab Pricing page](https://about.gitlab.com/pricing/). Additional minutes: - Are only used after the shared quota included in your subscription runs out. - Roll over month to month. diff --git a/doc/subscriptions/self_managed/index.md b/doc/subscriptions/self_managed/index.md index acdbdefc671..aee18e3d763 100644 --- a/doc/subscriptions/self_managed/index.md +++ b/doc/subscriptions/self_managed/index.md @@ -108,7 +108,7 @@ GitLab has several features which can help you manage the number of users: - Enable the [**Require administrator approval for new sign ups**](../../user/admin_area/settings/sign_up_restrictions.md#require-administrator-approval-for-new-sign-ups) option. -- Enable `block_auto_created_users` for new sign-ups via [LDAP](../../administration/auth/ldap/index.md#basic-configuration-settings) or [OmniAuth](../../integration/omniauth.md#initial-omniauth-configuration). +- Enable `block_auto_created_users` for new sign-ups via [LDAP](../../administration/auth/ldap/index.md#basic-configuration-settings) or [OmniAuth](../../integration/omniauth.md#configure-initial-settings). - Enable the [User cap](../../user/admin_area/settings/sign_up_restrictions.md#user-cap) option. **Available in GitLab 13.7 and later**. - [Disable new sign-ups](../../user/admin_area/settings/sign_up_restrictions.md), and instead manage new @@ -181,9 +181,9 @@ The daily job provides **only** the following information to the Customers Porta - Instance ID - MD5 hash of license -<details> -<summary>Click here to view an example of a cloud licensing sync request.</summary> -<pre><code> +Example of a cloud licensing sync request: + +```json { "gitlab_version": "14.1.0-pre", "timestamp": "2021-06-14T12:00:09Z", @@ -231,8 +231,7 @@ The daily job provides **only** the following information to the Customers Porta "instance_id": "9367590b-82ad-48cb-9da7-938134c29088", "license_md5": "002f02470fe45ef6a333a4282aca6222" } -</code></pre> -</details> +``` #### Sync subscription details diff --git a/doc/system_hooks/system_hooks.md b/doc/system_hooks/system_hooks.md index fbeee7b96bc..dcab56a0d0f 100644 --- a/doc/system_hooks/system_hooks.md +++ b/doc/system_hooks/system_hooks.md @@ -1,6 +1,6 @@ --- -stage: none -group: unassigned +stage: Ecosystem +group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments type: reference --- diff --git a/doc/topics/autodevops/customize.md b/doc/topics/autodevops/customize.md index c01ed4a49d0..906fea2e6ad 100644 --- a/doc/topics/autodevops/customize.md +++ b/doc/topics/autodevops/customize.md @@ -93,6 +93,28 @@ Avoid passing secrets as Docker build arguments if possible, as they may be persisted in your image. See [this discussion of best practices with secrets](https://github.com/moby/moby/issues/13490) for details. +## Custom container image + +By default, [Auto Deploy](stages.md#auto-deploy) deploys a container image built and pushed to the GitLab registry by [Auto Build](stages.md#auto-build). +You can override this behavior by defining specific variables: + +| Entry | Default | Can be overridden by | +| ----- | ----- | ----- | +| Image Path | `$CI_REGISTRY_IMAGE/$CI_COMMIT_REF_SLUG` for branch pipelines. `$CI_REGISTRY_IMAGE` for tag pipelines. | `$CI_APPLICATION_REPOSITORY` | +| Image Tag | `$CI_COMMIT_SHA` for branch pipelines. `$CI_COMMIT_TAG` for tag pipelines. | `$CI_APPLICATION_TAG` | + +These variables also affect Auto Build. If you don't want to build and push an image to +`$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG`, consider +including only `Jobs/Deploy.gitlab-ci.yml`, or [disabling the `build` jobs](#disable-jobs). + +Here is an example setup in your `.gitlab-ci.yml`: + +```yaml +variables: + CI_APPLICATION_REPOSITORY: <your-image-repository> + CI_APPLICATION_TAG: <the-tag> +``` + ## Extend Auto DevOps with the API You can extend and manage your Auto DevOps configuration with GitLab APIs: @@ -218,7 +240,7 @@ See [Multiple Kubernetes clusters for Auto DevOps](multiple_clusters_auto_devops For clusters not managed by GitLab, you can customize the namespace in `.gitlab-ci.yml` by specifying -[`environment:kubernetes:namespace`](../../ci/environments/index.md#configure-kubernetes-deployments). +[`environment:kubernetes:namespace`](../../ci/environments/index.md#configure-kubernetes-deployments-deprecated). For example, the following configuration overrides the namespace used for `production` deployments: @@ -392,6 +414,8 @@ applications. | `CANARY_ENABLED` | From GitLab 11.0, used to define a [deploy policy for canary environments](#deploy-policy-for-canary-environments). | | `CANARY_PRODUCTION_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md) in the production environment. Takes precedence over `CANARY_REPLICAS`. Defaults to 1. | | `CANARY_REPLICAS` | Number of canary replicas to deploy for [Canary Deployments](../../user/project/canary_deployments.md). Defaults to 1. | +| `CI_APPLICATION_REPOSITORY` | The repository of container image being built or deployed, `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG`. For more details, read [Custom container image](#custom-container-image). | +| `CI_APPLICATION_TAG` | The tag of the container image being built or deployed, `$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG`. For more details, read [Custom container image](#custom-container-image). | | `DAST_AUTO_DEPLOY_IMAGE_VERSION` | Customize the image version used for DAST deployments on the default branch. Should usually be the same as `AUTO_DEPLOY_IMAGE_VERSION`. See [list of versions](https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/releases). | | `DOCKERFILE_PATH` | From GitLab 13.2, allows overriding the [default Dockerfile path for the build stage](#custom-dockerfile) | | `HELM_RELEASE_NAME` | From GitLab 12.1, allows the `helm` release name to be overridden. Can be used to assign unique release names when deploying multiple projects to a single namespace. | @@ -399,7 +423,10 @@ applications. | `HELM_UPGRADE_EXTRA_ARGS` | From GitLab 11.11, allows extra options in `helm upgrade` commands when deploying the application. Note that using quotes doesn't prevent word splitting. | | `INCREMENTAL_ROLLOUT_MODE` | From GitLab 11.4, if present, can be used to enable an [incremental rollout](#incremental-rollout-to-production) of your application for the production environment. Set to `manual` for manual deployment jobs or `timed` for automatic rollout deployments with a 5 minute delay each one. | | `K8S_SECRET_*` | From GitLab 11.7, any variable prefixed with [`K8S_SECRET_`](#application-secret-variables) is made available by Auto DevOps as environment variables to the deployed application. | +| `KUBE_CONTEXT` | From GitLab 14.5, can be used to select which context to use from `KUBECONFIG`. When `KUBE_CONTEXT` is blank, the default context in `KUBECONFIG` (if any) will be used. A context must be selected when using the [CI/CD tunnel](../../user/clusters/agent/ci_cd_tunnel.md). | | `KUBE_INGRESS_BASE_DOMAIN` | From GitLab 11.8, can be used to set a domain per cluster. See [cluster domains](../../user/project/clusters/gitlab_managed_clusters.md#base-domain) for more information. | +| `KUBE_NAMESPACE` | The namespace used for deployments. When using certificate-based clusters, [this value should not be overwritten directly](../../user/project/clusters/deploy_to_cluster.md#custom-namespace). | +| `KUBECONFIG` | The kubeconfig to use for deployments. User-provided values take priority over GitLab-provided values. | | `PRODUCTION_REPLICAS` | Number of replicas to deploy in the production environment. Takes precedence over `REPLICAS` and defaults to 1. For zero downtime upgrades, set to 2 or greater. | | `REPLICAS` | Number of replicas to deploy. Defaults to 1. | | `ROLLOUT_RESOURCE_TYPE` | From GitLab 11.9, allows specification of the resource type being deployed when using a custom Helm chart. Default value is `deployment`. | diff --git a/doc/topics/autodevops/multiple_clusters_auto_devops.md b/doc/topics/autodevops/multiple_clusters_auto_devops.md index 2191ab9ee8b..c6df5ac9e02 100644 --- a/doc/topics/autodevops/multiple_clusters_auto_devops.md +++ b/doc/topics/autodevops/multiple_clusters_auto_devops.md @@ -35,8 +35,8 @@ To add a different cluster for each environment: 1. Navigate to your project's **Infrastructure > Kubernetes clusters**. 1. Create the Kubernetes clusters with their respective environment scope, as described from the table above. -1. After creating the clusters, navigate to each cluster and [install - Ingress](quick_start_guide.md#install-ingress). Wait for the Ingress IP address to be assigned. +1. After creating the clusters, navigate to each cluster and [install Ingress](quick_start_guide.md#install-ingress). + Wait for the Ingress IP address to be assigned. 1. Make sure you've [configured your DNS](requirements.md#auto-devops-base-domain) with the specified Auto DevOps domains. 1. Navigate to each cluster's page, through **Infrastructure > Kubernetes clusters**, diff --git a/doc/topics/autodevops/quick_start_guide.md b/doc/topics/autodevops/quick_start_guide.md index c84b5e4d9c7..1fc2bd7c2c0 100644 --- a/doc/topics/autodevops/quick_start_guide.md +++ b/doc/topics/autodevops/quick_start_guide.md @@ -236,7 +236,7 @@ you to common environment tasks: about the Kubernetes cluster and how the application affects it in terms of memory usage, CPU usage, and latency - **Deploy to** (**{play}** **{angle-down}**) - Displays a list of environments you can deploy to -- **Terminal** (**{terminal}**) - Opens a [web terminal](../../ci/environments/index.md#web-terminals) +- **Terminal** (**{terminal}**) - Opens a [web terminal](../../ci/environments/index.md#web-terminals-deprecated) session inside the container where the application is running - **Re-deploy to environment** (**{repeat}**) - For more information, see [Retrying and rolling back](../../ci/environments/index.md#retry-or-roll-back-a-deployment) diff --git a/doc/topics/autodevops/stages.md b/doc/topics/autodevops/stages.md index ead2e957684..039c369ce9b 100644 --- a/doc/topics/autodevops/stages.md +++ b/doc/topics/autodevops/stages.md @@ -663,7 +663,7 @@ may require commands to be wrapped as follows: Some of the reasons you may need to wrap commands: - Attaching using `kubectl exec`. -- Using the GitLab [Web Terminal](../../ci/environments/index.md#web-terminals). +- Using the GitLab [Web Terminal](../../ci/environments/index.md#web-terminals-deprecated). For example, to start a Rails console from the application root directory, run: diff --git a/doc/topics/git/getting_started.md b/doc/topics/git/getting_started.md index 7e04eae622f..7a836e5b659 100644 --- a/doc/topics/git/getting_started.md +++ b/doc/topics/git/getting_started.md @@ -21,6 +21,12 @@ comments: false git clone <url> ``` +NOTE: +You can also clone GitLab projects with the +[GitLab Workflow VS Code extension](../../user/project/repository/vscode.md). +To learn more, read about the extension's +[`Git: Clone` command](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#clone-gitlab-projects). + ## Central Repositories - To instantiate a central repository a `--bare` flag is required. diff --git a/doc/topics/git/git_rebase.md b/doc/topics/git/git_rebase.md index b09f9fa0f6c..c0bc7ed4e5c 100644 --- a/doc/topics/git/git_rebase.md +++ b/doc/topics/git/git_rebase.md @@ -3,13 +3,13 @@ stage: Create group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: concepts, howto -description: "Introduction to Git rebase, force-push, and resolving merge conflicts through the command line." +description: "Introduction to Git rebase and force-push, methods to resolve merge conflicts through the command line." --- -# Introduction to Git rebase, force-push, and merge conflicts **(FREE)** +# Introduction to Git rebase and force-push **(FREE)** This guide helps you to get started with rebasing, force-pushing, and fixing -merge conflicts locally. +[merge conflicts](../../user/project/merge_requests/conflicts.md) locally. Before diving into this document, make sure you are familiar with using [Git through the command line](../../gitlab-basics/start-using-git.md). @@ -26,7 +26,8 @@ Git. There are the following rebase options: WARNING: `git rebase` rewrites the commit history. It **can be harmful** to do it in -shared branches. It can cause complex and hard to resolve merge conflicts. In +shared branches. It can cause complex and hard to resolve +[merge conflicts](../../user/project/merge_requests/conflicts.md). In these cases, instead of rebasing your branch against the default branch, consider pulling it instead (`git pull origin master`). It has a similar effect without compromising the work of your contributors. @@ -117,7 +118,7 @@ example, `release-10-3`. You can also replace `origin` with other remote repositories, for example, `upstream`. To check what remotes you have linked to your local repository, you can run `git remote -v`. -If there are [merge conflicts](#merge-conflicts), Git prompts you to fix +If there are merge conflicts, Git prompts you to fix them before continuing the rebase. To learn more, check Git's documentation on [rebasing](https://git-scm.com/book/en/v2/Git-Branching-Rebasing) @@ -129,7 +130,7 @@ You can rebase your feature branch directly from the merge request through a [quick action](../../user/project/quick_actions.md#issues-merge-requests-and-epics), if all of these conditions are met: -- No [merge conflicts](#merge-conflicts) exist for your feature branch. +- No merge conflicts exist for your feature branch. - You have the **Developer** role for the source project. This role grants you permission to push to the source branch for the source project. - If the merge request is in a fork, the fork must allow commits @@ -144,6 +145,11 @@ To rebase from the UI: GitLab schedules a rebase of the feature branch against the default branch and executes it as soon as possible. +The user performing the rebase action is considered +a user that added commits to the merge request. When the merge request approvals setting +[**Prevent approvals by users who add commits**](../../user/project/merge_requests/approvals/settings.md#prevent-approvals-by-users-who-add-commits) +is enabled, this setting prevents the user from also approving the merge request. + ### Interactive rebase You can use interactive rebase to modify commits. For example, amend a commit @@ -235,70 +241,3 @@ you can't force push to it unless you either: to it. Then you can force push and protect it again. - -## Merge conflicts - -As Git is based on comparing versions of a file -line-by-line, whenever a line changed in your branch coincides with the same -line changed in the target branch (after the moment you created your feature branch from it), Git -identifies these changes as a merge conflict. To fix it, you need to choose -which version of that line you want to keep. - -Most conflicts can be [resolved through the GitLab UI](../../user/project/merge_requests/resolve_conflicts.md). - -For more complex cases, there are various methods for resolving them. There are -also [Git GUI apps](https://git-scm.com/downloads/guis) that can help by -visualizing the differences. - -To fix conflicts locally, you can use the following method: - -1. Open the terminal and checkout your feature branch, for example, `my-feature-branch`: - - ```shell - git checkout my-feature-branch - ``` - -1. [Rebase](#regular-rebase) your branch against the target branch so Git - prompts you with the conflicts: - - ```shell - git rebase origin/master - ``` - -1. Open the conflicting file in a code editor of your preference. -1. Look for the conflict block: - - It begins with the marker: `<<<<<<< HEAD`. - - Below, there is the content with your changes. - - The marker: `=======` indicates the end of your changes. - - Below, there's the content of the latest changes in the target branch. - - The marker `>>>>>>>` indicates the end of the conflict. -1. Edit the file: choose which version (before or after `=======`) you want to - keep, and then delete the portion of the content you don't want in the file. -1. Delete the markers. -1. Save the file. -1. Repeat the process if there are other conflicting files. -1. Stage your changes: - - ```shell - git add . - ``` - -1. Commit your changes: - - ```shell - git commit -m "Fix merge conflicts" - ``` - -1. Continue rebasing: - - ```shell - git rebase --continue - ``` - - WARNING: - Up to this point, you can run `git rebase --abort` to stop the process. - Git aborts the rebase and rolls back the branch to the state you had before - running `git rebase`. - After you run `git rebase --continue` the rebase **cannot** be aborted. - -1. [Force-push](#force-push) to your remote branch. diff --git a/doc/topics/git/lfs/index.md b/doc/topics/git/lfs/index.md index 0b4fd335455..b21237203d7 100644 --- a/doc/topics/git/lfs/index.md +++ b/doc/topics/git/lfs/index.md @@ -42,6 +42,7 @@ Documentation for GitLab instance administrators is under [LFS administration do credentials store is recommended. - Git LFS always assumes HTTPS so if you have GitLab server on HTTP you must [add the URL to Git configuration manually](#troubleshooting). +- [Group wikis](../../../user/project/wiki/group.md) do not support Git LFS. NOTE: With 8.12 GitLab added LFS support to SSH. The Git LFS communication diff --git a/doc/topics/index.md b/doc/topics/index.md index fde420c64f6..6d2c839430b 100644 --- a/doc/topics/index.md +++ b/doc/topics/index.md @@ -1,24 +1,9 @@ --- -stage: none -group: unassigned -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +redirect_to: '../index.md' +remove_date: '2022-02-03' --- -# Topics +This document was moved to [another location](../index.md). -Welcome to Topics! We have organized our content resources into topics -to get you started on areas of your interest. Each topic page -consists of an index listing all related content. It guides -you through better understanding GitLab concepts -through our regular docs, and, when available, through articles (guides, -tutorials, technical overviews, blog posts) and videos. - -- [Auto DevOps](autodevops/index.md) -- [Authentication](authentication/index.md) -- [Continuous Integration (GitLab CI/CD)](../ci/index.md) -- [Cron](cron/index.md) -- [Git](git/index.md) -- [GitLab Flow](gitlab_flow.md) -- [GitLab Installation](../install/index.md) -- [GitLab Pages](../user/project/pages/index.md) -- [Offline GitLab](offline/index.md) +<!-- This redirect file can be deleted after <2022-02-03>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/topics/plan_and_track.md b/doc/topics/plan_and_track.md index 86c5287b331..abede62c00b 100644 --- a/doc/topics/plan_and_track.md +++ b/doc/topics/plan_and_track.md @@ -31,6 +31,7 @@ Get work done as a team. - [Labels](../user/project/labels.md) - [Milestones](../user/project/milestones/index.md) - [Requirements](../user/project/requirements/index.md) +- [Tasks](../user/tasks.md) - [Time tracking](../user/project/time_tracking.md) - [Wikis](../user/project/wiki/index.md) diff --git a/doc/topics/release_your_application.md b/doc/topics/release_your_application.md index 31eb7705760..cbd7cfab720 100644 --- a/doc/topics/release_your_application.md +++ b/doc/topics/release_your_application.md @@ -4,10 +4,11 @@ group: info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Release your application **(FREE)** +# Deploy and release your application **(FREE)** -Release your application internally or to the public. Use +Deploy your application internally or to the public. Use flags to release features incrementally. +- [Environments and deployments](../ci/environments/index.md) - [Releases](../user/project/releases/index.md) - [Feature flags](../operations/feature_flags.md) diff --git a/doc/topics/use_gitlab.md b/doc/topics/use_gitlab.md index f45dc91131c..f73e41c251b 100644 --- a/doc/topics/use_gitlab.md +++ b/doc/topics/use_gitlab.md @@ -16,4 +16,6 @@ organize your work, create and secure your application, and analyze its performa - [Secure your application](../user/application_security/index.md) - [Release your application](release_your_application.md) - [Monitor application performance](../operations/index.md) +- [Monitor runner performance](https://docs.gitlab.com/runner/monitoring/index.html) +- [Manage your infrastructure](../user/infrastructure/index.md) - [Analyze GitLab usage](../user/analytics/index.md) diff --git a/doc/update/deprecations.md b/doc/update/deprecations.md index e2af4f453c0..42bd41ed74b 100644 --- a/doc/update/deprecations.md +++ b/doc/update/deprecations.md @@ -17,16 +17,26 @@ sole discretion of GitLab Inc. <!-- vale off --> <!-- +DO NOT EDIT THIS PAGE DIRECTLY + This page is automatically generated from the YAML files in `/data/deprecations` by the rake task located at `lib/tasks/gitlab/docs/compile_deprecations.rake`. -Do not edit this page directly. +For deprecation authors (usually Product Managers and Engineering Managers): + +- To add a deprecation, use the example.yml file in `/data/deprecations/templates` as a template. +- For more information about authoring deprecations, check the the deprecation item guidance: + https://about.gitlab.com/handbook/marketing/blog/release-posts/#creating-a-deprecation-entry + +For deprecation reviewers (Technical Writers only): -To add a deprecation, use the example.yml file in `/data/deprecations/templates` as a template, -then run `bin/rake gitlab:docs:compile_deprecations`. +- To update the deprecation doc, run: `bin/rake gitlab:docs:compile_deprecations` +- To verify the deprecations doc is up to date, run: `bin/rake gitlab:docs:check_deprecations` +- For more information about updating the deprecation doc, see the deprecation doc update guidance: + https://about.gitlab.com/handbook/marketing/blog/release-posts/#update-the-deprecations-doc --> -## 14.4 +## 14.5 ### Rename Task Runner pod to Toolbox @@ -44,17 +54,17 @@ The [release-cli](https://gitlab.com/gitlab-org/release-cli) will be released as Announced: 2021-08-22 -## 15.0 +## 14.8 -### Legacy database configuration +### openSUSE Leap 15.2 packages -The syntax of [GitLabs database](https://docs.gitlab.com/omnibus/settings/database.html) -configuration located in `database.yml` is changing and the legacy format is deprecated. The legacy format -supported using a single PostgreSQL adapter, whereas the new format is changing to support multiple databases. The `main:` database needs to be defined as a first configuration item. +Distribution support and security updates for openSUSE Leap 15.2 are [ending December 2021](https://en.opensuse.org/Lifetime#openSUSE_Leap). -This deprecation mainly impacts users compiling GitLab from source because Omnibus will handle this configuration automatically. +Starting in 14.5 we are providing packages for openSUSE Leap 15.3, and will stop providing packages for openSUSE Leap 15.2 in the 14.8 milestone. -Announced: 2021-09-22 +Announced: 2021-11-22 + +## 15.0 ### Audit events for repository push events @@ -66,15 +76,39 @@ dramatically slow down GitLab instances. For this reason, they are being removed Announced: 2021-09-22 -### OmniAuth Kerberos gem +### Certificate-based integration with Kubernetes -The `omniauth-kerberos` gem will be removed in our next major release, GitLab 15.0. +[We are deprecating the certificate-based integration with Kubernetes](https://about.gitlab.com/blog/2021/11/15/deprecating-the-cert-based-kubernetes-integration/). +The timeline of removal of the integration from the product is not yet planned and we will communicate +more details as they emerge. The certificate-based integration will continue to receive security and +critical fixes, and features built on the integration will continue to work with supported Kubernetes +versions. We will provide migration plans in a future iteration. See [the list of features affected by this deprecation](https://docs.gitlab.com/ee/user/infrastructure/clusters/#deprecated-features). +For updates and details, follow this [epic](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). -This gem has not been maintained and has very little usage. We therefore plan to remove support for this authentication method and recommend using the Kerberos [SPNEGO](https://en.wikipedia.org/wiki/SPNEGO) integration instead. You can follow the [upgrade instructions](https://docs.gitlab.com/ee/integration/kerberos.html#upgrading-from-password-based-to-ticket-based-kerberos-sign-ins) to upgrade from the `omniauth-kerberos` integration to the supported one. +For a more robust, secure, forthcoming, and reliable integration with Kubernetes, we recommend the use of the +[Kubernetes Agent](https://docs.gitlab.com/ee/user/clusters/agent/) to connect Kubernetes clusters with GitLab. -Note that we are not deprecating the Kerberos SPNEGO integration, only the old password-based Kerberos integration. +Announced: 2021-11-15 -Announced: 2021-09-22 +### Converting an instance (shared) runner to a project (specific) runner is deprecated + +In GitLab 15.0, we will remove the feature that enables you to convert an instance (shared) runner to a project (specific) runner. Users who need to add a runner to only a particular project can register a runner to the project directly. + +Announced: 2021-11-22 + +### Deprecate `Versions` on base `PackageType` + +As part of the work to create a [Package Registry GraphQL API](https://gitlab.com/groups/gitlab-org/-/epics/6318), the Package group deprecated the `Version` type for the basic `PackageType` type and moved it to [`PackageDetailsType`](https://docs.gitlab.com/ee/api/graphql/reference/index.html#packagedetailstype). + +In milestone 15.0, we will completely remove `Version` from `PackageType`. + +Announced: 2021-11-22 + +### Deprecate support for SLES 12 SP2 + +Long term service and support (LTSS) for SUSE Linux Enterprise Server (SLES) 12 SP2 [ended on March 31, 2021](https://www.suse.com/lifecycle/). The CA certificates on SP2 include the expired DST root certificate, and it's not getting new CA certificate package updates. We have implemented some [workarounds](https://gitlab.com/gitlab-org/gitlab-omnibus-builder/-/merge_requests/191), but we will not be able to continue to keep the build running properly. + +Announced: 2021-11-22 ### GitLab Serverless @@ -84,7 +118,23 @@ We decided to remove the GitLab Serverless features as they never really resonat Announced: 2021-09-22 -## 15.2 +### Known host required for GitLab Runner SSH executor + +In [GitLab 14.3](https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/3074), we added a configuration setting in the GitLab Runner `config.toml` file. This setting, [`[runners.ssh.disable_strict_host_key_checking]`](https://docs.gitlab.com/runner/executors/ssh.html#security), controls whether or not to use strict host key checking with the SSH executor. + +In GitLab 15.0 and later, the default value for this configuration option will change from `true` to `false`. This means that strict host key checking will be enforced when using the GitLab Runner SSH executor. + +Announced: 2021-11-22 + +### Legacy database configuration + +The syntax of [GitLabs database](https://docs.gitlab.com/omnibus/settings/database.html) +configuration located in `database.yml` is changing and the legacy format is deprecated. The legacy format +supported using a single PostgreSQL adapter, whereas the new format is changing to support multiple databases. The `main:` database needs to be defined as a first configuration item. + +This deprecation mainly impacts users compiling GitLab from source because Omnibus will handle this configuration automatically. + +Announced: 2021-09-22 ### NFS for Git repository storage deprecated @@ -99,3 +149,95 @@ Gitaly Cluster offers tremendous benefits for our customers such as: We encourage customers currently using NFS for Git repositories to plan their migration by reviewing our documentation on [migrating to Gitaly Cluster](https://docs.gitlab.com/ee/administration/gitaly/index.html#migrate-to-gitaly-cluster). Announced: 2021-06-22 + +### OmniAuth Kerberos gem + +The `omniauth-kerberos` gem will be removed in our next major release, GitLab 15.0. + +This gem has not been maintained and has very little usage. We therefore plan to remove support for this authentication method and recommend using the Kerberos [SPNEGO](https://en.wikipedia.org/wiki/SPNEGO) integration instead. You can follow the [upgrade instructions](https://docs.gitlab.com/ee/integration/kerberos.html#upgrading-from-password-based-to-ticket-based-kerberos-sign-ins) to upgrade from the `omniauth-kerberos` integration to the supported one. + +Note that we are not deprecating the Kerberos SPNEGO integration, only the old password-based Kerberos integration. + +Announced: 2021-09-22 + +### Package pipelines in API payload is paginated + +A request to the API for `/api/v4/projects/:id/packages` returns a paginated result of packages. Each package lists all of its pipelines in this response. This is a performance concern, as it's possible for a package to have hundreds or thousands of associated pipelines. + +In milestone 15.0, we will remove the `pipelines` attribute from the API response. + +Announced: 2021-11-22 + +### REST API Runner will not contain `paused` + +Runner REST API will not return `paused` as a status in GitLab 15.0. + +Paused runners' status will only relate to runner contact status, such as: +`online`, `offline`, or `not_connected`. Status `paused` will not appear when the runner is +not active. + +When checking if a runner is `paused`, API users are advised to check the boolean attribute +`active` to be `false` instead. + +Announced: 2021-11-22 + +### Removal of `promote-db` command from `gitlab-ctl` + +In GitLab 14.5, we introduced the command `gitlab-ctl promote` to promote any Geo secondary node to a primary during a failover. This command replaces `gitlab-ctl promote-db` which is used to promote database nodes in multi-node Geo secondary sites. `gitlab-ctl promote-db` will continue to function as-is and be available until GitLab 15.0. We recommend that Geo customers begin testing the new `gitlab-ctl promote` command in their staging environments and incorporating the new command in their failover procedures. + +Announced: 2021-11-22 + +### Removal of `promote-to-primary-node` command from `gitlab-ctl` + +In GitLab 14.5, we introduced the command `gitlab-ctl promote` to promote any Geo secondary node to a primary during a failover. This command replaces `gitlab-ctl promote-to-primary-node` which was only usable for single-node Geo sites. `gitlab-ctl promote-to-primary-node` will continue to function as-is and be available until GitLab 15.0. We recommend that Geo customers begin testing the new `gitlab-ctl promote` command in their staging environments and incorporating the new command in their failover procedures. + +Announced: 2021-11-22 + +### Remove the `:dependency_proxy_for_private_groups` feature flag + +We added a feature flag because [GitLab-#11582](https://gitlab.com/gitlab-org/gitlab/-/issues/11582) changed how public groups use the Dependency Proxy. Prior to this change, you could use the Dependency Proxy without authentication. The change requires authentication to use the Dependency Proxy. + +In milestone 15.0, we will remove the feature flag entirely. Moving forward, you must authenticate when using the Dependency Proxy. + +Announced: 2021-11-22 + +### Remove the `pipelines` field from the `version` field + +In GraphQL, there are two `pipelines` fields that you can use in a [`PackageDetailsType`](https://docs.gitlab.com/ee/api/graphql/reference/#packagedetailstype) to get the pipelines for package versions: + +- The `versions` field's `pipelines` field. This returns all the pipelines associated with all the package's versions, which can pull an unbounded number of objects in memory and create performance concerns. +- The `pipelines` field of a specific `version`. This returns only the pipelines associated with that single package version. + +To mitigate possible performance problems, we will remove the `versions` field's `pipelines` field in milestone 15.0. Although you will no longer be able to get all pipelines for all versions of a package, you can still get the pipelines of a single version through the remaining `pipelines` field for that version. + +Announced: 2021-11-22 + +### Update to the Container Registry group-level API + +In milestone 15.0, support for the `tags` and `tags_count` parameters will be removed from the Container Registry API that [gets registry repositories from a group](../api/container_registry.md#within-a-group). + +The `GET /groups/:id/registry/repositories` endpoint will remain, but won't return any info about tags. To get the info about tags, you can use the existing `GET /registry/repositories/:id` endpoint, which will continue to support the `tags` and `tag_count` options as it does today. The latter must be called once per image repository. + +Announced: 2021-11-22 + +### Value Stream Analytics filtering calculation change + +We are changing how the date filter works in Value Stream Analytics. Instead of filtering by the time that the issue or merge request was created, the date filter will filter by the end event time of the given stage. This will result in completely different figures after this change has rolled out. + +If you monitor Value Stream Analytics metrics and rely on the date filter, to avoid losing data, you must save the data prior to this change. + +Announced: 2021-11-22 + +### `AuthenticationType` for `[runners.cache.s3]` must be explicitly assigned + +In GitLab 15.0 and later, to access the AWS S3 cache, you must specify the `AuthenticationType` for [`[runners.cache.s3]`](https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runnerscaches3-section). The `AuthenticationType` must be `IAM` or `credentials`. + +Prior to 14.5, if you did not define the `AuthenticationType`, GitLab Runner chose a type for you. + +Announced: 2021-11-22 + +### defaultMergeCommitMessageWithDescription GraphQL API field will be removed in GitLab 15.0 + +The GraphQL API field `defaultMergeCommitMessageWithDescription` has been deprecated and will be removed in GitLab 15.0. For projects with a commit message template set, it will ignore the template. + +Announced: 2021-11-22 diff --git a/doc/update/index.md b/doc/update/index.md index b719c47ae26..bb7fdaa93c0 100644 --- a/doc/update/index.md +++ b/doc/update/index.md @@ -12,6 +12,11 @@ GitLab version is, if you're upgrading to a major version, and so on. Make sure to read the whole page as it contains information related to every upgrade method. +NOTE: +Upgrade GitLab to the latest available patch release, for example `13.8.8` rather than `13.8.0`. +This includes [versions you must stop at on the upgrade path](#upgrade-paths) as there may +be fixes for issues relating to the upgrade process. + The [maintenance policy documentation](../policy/maintenance.md) has additional information about upgrading, including: @@ -76,22 +81,44 @@ See the guide to [plan your GitLab upgrade](plan_your_upgrade.md). ## Checking for background migrations before upgrading -Certain major/minor releases may require different migrations to be -finished before you update to the newer version. +Certain releases may require different migrations to be +finished before you update to the newer version. Additionally check +[batched migrations](#batched-background-migrations) from GitLab 14.0. Decrease the time required to complete these migrations by increasing the number of [Sidekiq workers](../administration/operations/extra_sidekiq_processes.md) that can process jobs in the `background_migration` queue. -**For GitLab 14.0 and newer** +**For Omnibus installations:** + +```shell +sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +``` + +**For installations from source:** + +```shell +cd /home/git/gitlab +sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +``` + +### Batched background migrations + +GitLab 14.0 introduced [batched background migrations](../user/admin_area/monitoring/background_migrations.md). -To check the status of [batched background migrations](../user/admin_area/monitoring/background_migrations.md): +Some installations [may need to run GitLab 14.0 for at least a day](#1400) to complete the database changes introduced by that upgrade. + +#### Check the status of batched background migrations + +To check the status of batched background migrations: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Monitoring > Background Migrations**.  +All migrations must have a `Finished` status before you upgrade GitLab. + The status of batched background migrations can also be queried directly in the database. 1. Log into a `psql` prompt according to the directions for your instance's installation method @@ -102,20 +129,14 @@ The status of batched background migrations can also be queried directly in the select job_class_name, table_name, column_name, job_arguments from batched_background_migrations where status <> 3; ``` -**For Omnibus installations** +If the migrations are not finished and you try to update to a later version, +GitLab prompts you with an error: -You can also run: - -```shell -sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' +```plaintext +Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': ``` -**For installations from source** - -```shell -cd /home/git/gitlab -sudo -u git -H bundle exec rails runner -e production 'puts Gitlab::BackgroundMigration.remaining' -``` +If you get this error, [check the batched background migration options](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished) to complete the upgrade. ### What do I do if my background migrations are stuck? @@ -148,6 +169,10 @@ pending_job_classes = scheduled_queue.select { |job| job["class"] == "Background pending_job_classes.each { |job_class| Gitlab::BackgroundMigration.steal(job_class) } ``` +**Batched migrations (GitLab 14.0 and newer):** + +See [troubleshooting batched background migrations](../user/admin_area/monitoring/background_migrations.md#troubleshooting). + ## Dealing with running CI/CD pipelines and jobs If you upgrade your GitLab instance while the GitLab Runner is processing jobs, the trace updates fail. When GitLab is back online, the trace updates should self-heal. However, depending on the error, the GitLab Runner either retries or eventually terminates job handling. @@ -163,12 +188,10 @@ To address the above two scenario's, it is advised to do the following prior to ## Checking for pending Advanced Search migrations -This section is only applicable if you have enabled the [Elasticsearch -integration](../integration/elasticsearch.md). +This section is only applicable if you have enabled the [Elasticsearch integration](../integration/elasticsearch.md). -Major releases require all [Advanced Search -migrations](../integration/elasticsearch.md#advanced-search-migrations) to -be finished from the most recent minor release in your current version +Major releases require all [Advanced Search migrations](../integration/elasticsearch.md#advanced-search-migrations) +to be finished from the most recent minor release in your current version before the major version upgrade. You can find pending migrations by running the following command: @@ -187,8 +210,7 @@ sudo -u git -H bundle exec rake gitlab:elastic:list_pending_migrations ### What do I do if my Advanced Search migrations are stuck? -See [how to retry a halted -migration](../integration/elasticsearch.md#retry-a-halted-migration). +See [how to retry a halted migration](../integration/elasticsearch.md#retry-a-halted-migration). ## Upgrade paths @@ -200,7 +222,7 @@ Find where your version sits in the upgrade path below, and upgrade GitLab accordingly, while also consulting the [version-specific upgrade instructions](#version-specific-upgrading-instructions): -`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [latest `13.12.Z`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) +`8.11.Z` -> `8.12.0` -> `8.17.7` -> `9.5.10` -> `10.8.7` -> [`11.11.8`](#1200) -> `12.0.12` -> [`12.1.17`](#1210) -> `12.10.14` -> `13.0.14` -> [`13.1.11`](#1310) -> [`13.8.8`](#1388) -> [`13.12.15`](https://about.gitlab.com/releases/categories/releases/) -> [latest `14.0.Z`](#1400) -> [latest `14.1.Z`](#1410) -> [latest `14.Y.Z`](https://about.gitlab.com/releases/categories/releases/) The following table, while not exhaustive, shows some examples of the supported upgrade paths. @@ -234,12 +256,11 @@ It's also important to ensure that any background migrations have been fully com before upgrading to a new major version. To see the current size of the `background_migration` queue, [Check for background migrations before upgrading](#checking-for-background-migrations-before-upgrading). -If you have enabled the [Elasticsearch -integration](../integration/elasticsearch.md), then ensure +If you have enabled the [Elasticsearch integration](../integration/elasticsearch.md), then ensure all Advanced Search migrations are completed in the last minor version within -your current version. Be sure to [check for pending Advanced Search -migrations](#checking-for-pending-advanced-search-migrations) before proceeding -with the major version upgrade. +your current version. Be sure to +[check for pending Advanced Search migrations](#checking-for-pending-advanced-search-migrations) +before proceeding with the major version upgrade. If your GitLab instance has any runners associated with it, it is very important to upgrade GitLab Runner to match the GitLab minor version that was @@ -307,6 +328,33 @@ NOTE: Specific information that follow related to Ruby and Git versions do not apply to [Omnibus installations](https://docs.gitlab.com/omnibus/) and [Helm Chart deployments](https://docs.gitlab.com/charts/). They come with appropriate Ruby and Git versions and are not using system binaries for Ruby and Git. There is no need to install Ruby or Git when utilizing these two approaches. +### 14.5.0 + +- When `make` is run, Gitaly builds are now created in `_build/bin` and no longer in the root directory of the source directory. If you +are using a source install, update paths to these binaries in your [systemd unit files](upgrading_from_source.md#configure-systemd-units) +or [init scripts](upgrading_from_source.md#configure-sysv-init-script) by [following the documentation](upgrading_from_source.md). + +- Connections between Workhorse and Gitaly use the Gitaly `backchannel` protocol by default. If you deployed a gRPC proxy between Workhorse and Gitaly, + Workhorse can no longer connect. As a workaround, [disable the temporary `workhorse_use_sidechannel`](../administration/feature_flags.md#enable-or-disable-the-feature) + feature flag. If you need a proxy between Workhorse and Gitaly, use a TCP proxy. If you have feedback about this change, please go to [this issue](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/1301). + +- In 14.1 we introduced a background migration that changes how we store merge request diff commits + in order to significantly reduce the amount of storage needed. + In 14.5 we introduce a set of migrations that wrap up this process by making sure + that all remaining jobs over the `merge_request_diff_commits` table are completed. + These jobs will have already been processed in most cases so that no extra time is necessary during an upgrade to 14.5. + But if there are remaining jobs, the deployment may take a few extra minutes to complete. + + All merge request diff commits will automatically incorporate these changes, and there are no + additional requirements to perform the upgrade. + Existing data in the `merge_request_diff_commits` table remains unpacked until you run `VACUUM FULL merge_request_diff_commits`. + But note that the `VACUUM FULL` operation locks and rewrites the entire `merge_request_diff_commits` table, + so the operation takes some time to complete and it blocks access to this table until the end of the process. + We advise you to only run this command while GitLab is not actively used or it is taken offline for the duration of the process. + The time it takes to complete depends on the size of the table, which can be obtained by using `select pg_size_pretty(pg_total_relation_size('merge_request_diff_commits'));`. + + For more information, refer to [this issue](https://gitlab.com/gitlab-org/gitlab/-/issues/331823). + ### 14.4.0 Git 2.33.x and later is required. We recommend you use the @@ -314,9 +362,11 @@ Git 2.33.x and later is required. We recommend you use the ### 14.3.0 -Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). +- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading + to 14.3.Z from earlier GitLab 14 releases. +- Ruby 2.7.4 is required. Refer to [the Ruby installation instructions](../install/installation.md#2-ruby) for how to proceed. - - GitLab 14.3.0 contains post-deployment migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - [`ci_builds.id`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/70245) @@ -337,13 +387,10 @@ for how to proceed. ### 14.2.0 -- Due to an issue where `BatchedBackgroundMigrationWorkers` were - [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) - for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need - to update to at least 14.1.0 that contains the same fix before you update to 14.2. +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases). +- Ensure [batched background migrations finish](#batched-background-migrations) before upgrading + to 14.2.Z from earlier GitLab 14 releases. - GitLab 14.2.0 contains background migrations to [address Primary Key overflow risk for tables with an integer PK](https://gitlab.com/groups/gitlab-org/-/epics/4785) for the tables listed below: - - [`ci_build_needs`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65216) - [`ci_build_trace_chunks`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66123) - [`ci_builds_runner_session`](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/66433) @@ -366,35 +413,24 @@ for how to proceed. ### 14.1.0 -- Due to an issue where `BatchedBackgroundMigrationWorkers` were - [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) - for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - and a [14.0.Z](#1400) version was released. If you haven't updated to 14.0.5, you need - to update to at least 14.1.0 that contains the same fix before you update to - a later version. - - After you update to 14.1.0, - [batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations) - before you update to a later version. - - If the migrations are not finished and you try to update to a later version, - you'll see an error like: - - ```plaintext - Expected batched background migration for the given configuration to be marked as 'finished', but it is 'active': - ``` - - See how to [resolve this error](../user/admin_area/monitoring/background_migrations.md#database-migrations-failing-because-of-batched-background-migration-not-finished). +- [Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later](#upgrading-to-later-14y-releases) + but can upgrade to 14.1.Z. +- It is not required for instances already running 14.0.5 (or higher) to stop at 14.1.Z. + 14.1 is included on the upgrade path for the broadest compatibility + with self-managed installations, and ensure 14.0.0-14.0.4 installations do not + encounter issues with [batched background migrations](#batched-background-migrations). ### 14.0.0 +- Database changes made by the upgrade to GitLab 14.0 can take hours or days to complete on larger GitLab instances. + These [batched background migrations](#batched-background-migrations) update whole database tables to mitigate primary key overflow and must be finished before upgrading to GitLab 14.2 or higher. - Due to an issue where `BatchedBackgroundMigrationWorkers` were [not working](https://gitlab.com/gitlab-org/charts/gitlab/-/issues/2785#note_614738345) for self-managed instances, a [fix was created](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/65106) - that requires an update to at least 14.0.5. + that requires an update to at least 14.0.5. The fix was also released in [14.1.0](#1410). After you update to 14.0.5 or a later 14.0 patch version, - [batched background migrations need to finish](../user/admin_area/monitoring/background_migrations.md#check-the-status-of-background-migrations) + [batched background migrations need to finish](#batched-background-migrations) before you update to a later version. If the migrations are not finished and you try to update to a later version, @@ -413,6 +449,16 @@ for how to proceed. You should instead follow a [supported upgrade path](#upgrade-paths). - The support of PostgreSQL 11 [has been dropped](../install/requirements.md#database). Make sure to [update your database](https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server) to version 12 before updating to GitLab 14.0. +#### Upgrading to later 14.Y releases + +- Instances running 14.0.0 - 14.0.4 should not upgrade directly to GitLab 14.2 or later, + because of [batched background migrations](#batched-background-migrations). + 1. Upgrade first to either: + - 14.0.5 or a later 14.0.Z patch release. + - 14.1.0 or a later 14.1.Z patch release. + 1. [Batched background migrations need to finish](#batched-background-migrations) + before you update to a later version [and may take longer than usual](#1400). + ### 13.11.0 Git 2.31.x and later is required. We recommend you use the diff --git a/doc/update/package/index.md b/doc/update/package/index.md index 776a7111188..27845caed76 100644 --- a/doc/update/package/index.md +++ b/doc/update/package/index.md @@ -262,21 +262,36 @@ unable to add the `commit_message_regex_change` column. This results in the [backport migration of EE tables](https://gitlab.com/gitlab-org/gitlab/-/blob/cf00e431024018ddd82158f8a9210f113d0f4dbc/db/migrate/20190402150158_backport_enterprise_schema.rb#L1619) not working correctly. The backport migration assumes that certain tables in the database do not exist when running CE. -To fix this issue, manually add the missing `commit_message_negative_regex` column and restart GitLab: +To fix this issue: -```shell -# Access psql -sudo gitlab-rails dbconsole +1. Start a database console: -# Add the missing column -ALTER TABLE push_rules ADD COLUMN commit_message_negative_regex VARCHAR; + In GitLab 14.2 and later: -# Exit psql -\q + ```shell + sudo gitlab-rails dbconsole --database main + ``` -# Restart GitLab -sudo gitlab-ctl restart -``` + In GitLab 14.1 and earlier: + + ```shell + sudo gitlab-rails dbconsole + ``` + +1. Manually add the missing `commit_message_negative_regex` column: + + ```sql + ALTER TABLE push_rules ADD COLUMN commit_message_negative_regex VARCHAR; + + # Exit psql + \q + ``` + +1. Restart GitLab: + + ```shell + sudo gitlab-ctl restart + ``` ### Error `Failed to connect to the internal GitLab API` on a separate GitLab Pages server diff --git a/doc/update/patch_versions.md b/doc/update/patch_versions.md index d09f19d143b..a2d672e00ac 100644 --- a/doc/update/patch_versions.md +++ b/doc/update/patch_versions.md @@ -20,6 +20,10 @@ It's useful to make a backup just in case things go south. Depending on the inst ### 1. Stop server ```shell +# For systems running systemd +sudo systemctl stop gitlab.target + +# For systems running SysV init sudo service gitlab stop ``` @@ -108,6 +112,11 @@ Please follow the [install instruction](../integration/elasticsearch.md#install- ### 9. Start application ```shell +# For systems running systemd +sudo systemctl start gitlab.target +sudo systemctl restart nginx.service + +# For systems running SysV init sudo service gitlab start sudo service nginx restart ``` diff --git a/doc/update/upgrading_from_source.md b/doc/update/upgrading_from_source.md index d882de62c06..4343b464ba6 100644 --- a/doc/update/upgrading_from_source.md +++ b/doc/update/upgrading_from_source.md @@ -54,6 +54,10 @@ sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production ### 2. Stop server ```shell +# For systems running systemd +sudo systemctl stop gitlab.target + +# For systems running SysV init sudo service gitlab stop ``` @@ -229,7 +233,29 @@ ActionMailer::Base.delivery_method = :smtp See [`smtp_settings.rb.sample`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/smtp_settings.rb.sample#L13) as an example. -#### Init script +#### Configure systemd units + +If using the SysV init script, see [Configure SysV init script](#configure-sysv-init-script). + +Check if the systemd units have been updated: + +```shell +cd /home/git/gitlab + +git diff origin/PREVIOUS_BRANCH:lib/support/systemd origin/BRANCH:lib/support/systemd +``` + +Copy them over: + +```shell +sudo mkdir -p /usr/local/lib/systemd/system +sudo cp lib/support/systemd/* /usr/local/lib/systemd/system/ +sudo systemctl daemon-reload +``` + +#### Configure SysV init script + +If using systemd units, see [Configure systemd units](#configure-systemd-units). There might be new configuration options available for [`gitlab.default.example`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/support/init.d/gitlab.default.example). @@ -249,7 +275,7 @@ cd /home/git/gitlab sudo cp lib/support/init.d/gitlab /etc/init.d/gitlab ``` -For Ubuntu 16.04.1 LTS: +If you are using the init script on a system running systemd as init, because you have not switched to native systemd units yet, run: ```shell sudo systemctl daemon-reload @@ -341,6 +367,11 @@ sudo -u git -H make ### 15. Start application ```shell +# For systems running systemd +sudo systemctl start gitlab.target +sudo systemctl restart nginx.service + +# For systems running SysV init sudo service gitlab start sudo service nginx restart ``` diff --git a/doc/update/zero_downtime.md b/doc/update/zero_downtime.md index 7a74435267f..a311731cadd 100644 --- a/doc/update/zero_downtime.md +++ b/doc/update/zero_downtime.md @@ -1,17 +1,17 @@ --- stage: Enablement group: Distribution -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- # Zero downtime upgrades **(FREE SELF)** -Starting with GitLab 9.1.0 it's possible to upgrade to a newer major, minor, or -patch version of GitLab without having to take your GitLab instance offline. -However, for this to work there are the following requirements: +It's possible to upgrade to a newer major, minor, or patch version of GitLab +without having to take your GitLab instance offline. However, for this to work +there are the following requirements: -- You can only upgrade 1 minor release at a time. So from 9.1 to 9.2, not to - 9.3. If you skip releases, database modifications may be run in the wrong +- You can only upgrade one minor release at a time. So from 13.1 to 13.2, not to + 13.3. If you skip releases, database modifications may be run in the wrong sequence [and leave the database schema in a broken state](https://gitlab.com/gitlab-org/gitlab/-/issues/321542). - You have to use [post-deployment migrations](../development/post_deployment_migrations.md). - You are using PostgreSQL. Starting from GitLab 12.1, MySQL is not supported. @@ -36,10 +36,10 @@ to re-read any database changes that have been made by post-deployment migration Most of the time you can safely upgrade from a patch release to the next minor release if the patch release is not the latest. For example, upgrading from -9.1.1 to 9.2.0 should be safe even if 9.1.2 has been released. We do recommend +14.1.1 to 14.2.0 should be safe even if 14.1.2 has been released. We do recommend you check the release posts of any releases between your current and target version just in case they include any migrations that may require you to upgrade -1 release at a time. +one release at a time. Some releases may also include so called "background migrations". These migrations are performed in the background by Sidekiq and are often used for @@ -63,21 +63,21 @@ the migrations that are being performed. To help explain this, let's look at some examples: -**Example 1:** You are running a large GitLab installation using version 9.4.2, -which is the latest patch release of 9.4. When GitLab 9.5.0 is released this -installation can be safely upgraded to 9.5.0 without requiring downtime if the -requirements mentioned above are met. You can also skip 9.5.0 and upgrade to -9.5.1 after it's released, but you **can not** upgrade straight to 9.6.0; you -_have_ to first upgrade to a 9.5.Z release. - -**Example 2:** You are running a large GitLab installation using version 9.4.2, -which is the latest patch release of 9.4. GitLab 9.5 includes some background -migrations, and 10.0 requires these to be completed (processing any -remaining jobs for you). Skipping 9.5 is not possible without downtime, and due +**Example 1:** You are running a large GitLab installation using version 13.4.2, +which is the latest patch release of 13.4. When GitLab 13.5.0 is released this +installation can be safely upgraded to 13.5.0 without requiring downtime if the +requirements mentioned above are met. You can also skip 13.5.0 and upgrade to +13.5.1 after it's released, but you **can not** upgrade straight to 13.6.0; you +_have_ to first upgrade to a 13.5.Z release. + +**Example 2:** You are running a large GitLab installation using version 13.4.2, +which is the latest patch release of 13.4. GitLab 13.5 includes some background +migrations, and 14.0 requires these to be completed (processing any +remaining jobs for you). Skipping 13.5 is not possible without downtime, and due to the background migrations would require potentially hours of downtime depending on how long it takes for the background migrations to complete. To -work around this you have to upgrade to 9.5.Z first, then wait at least a -week before upgrading to 10.0. +work around this you have to upgrade to 13.5.Z first, then wait at least a +week before upgrading to 14.0. **Example 3:** You use MySQL as the database for GitLab. Any upgrade to a new major/minor release requires downtime. If a release includes any background @@ -89,7 +89,7 @@ meet the other online upgrade requirements mentioned above. Before following these instructions, note the following **important** information: -- You can only upgrade 1 minor release at a time. So from 13.6 to 13.7, not to 13.8. +- You can only upgrade one minor release at a time. So from 13.6 to 13.7, not to 13.8. If you attempt more than one minor release, the upgrade may fail. - On single-node Omnibus deployments, updates with no downtime are not possible when using Puma because Puma always requires a complete restart. This is because the @@ -100,6 +100,8 @@ Before following these instructions, note the following **important** informatio these instructions, **it is not possible to always achieve true zero downtime updates**. Users may see some connections timeout or be refused for a few minutes, depending on which services need to restart. +- On Omnibus deployments, the `/etc/gitlab/gitlab.rb` configuration file must **not** have + `gitlab_rails['auto_migrate'] = true`. 1. Create an empty file at `/etc/gitlab/skip-auto-reconfigure`. This prevents upgrades from running `gitlab-ctl reconfigure`, which by default automatically stops GitLab, runs all database migrations, and restarts GitLab. @@ -156,7 +158,7 @@ you've completed these steps. ## Multi-node / HA deployment -You can only upgrade 1 minor release at a time. So from 13.6 to 13.7, not to 13.8. +You can only upgrade one minor release at a time. So from 13.6 to 13.7, not to 13.8. If you attempt more than one minor release, the upgrade may fail. ### Use a load balancer in front of web (Puma) nodes @@ -208,7 +210,9 @@ load balancer to latest GitLab version. If you are an Enterprise Edition user, replace `gitlab-ce` with `gitlab-ee` in the above command. - 1. Get the regular migrations and latest code in place: + 1. Get the regular migrations and latest code in place. Before running this step, + the deploy node's `/etc/gitlab/gitlab.rb` configuration file must have + `gitlab_rails['auto_migrate'] = true` to permit regular migrations. ```shell sudo SKIP_POST_DEPLOYMENT_MIGRATIONS=true gitlab-ctl reconfigure diff --git a/doc/user/admin_area/analytics/dev_ops_report.md b/doc/user/admin_area/analytics/dev_ops_report.md index b5294bb265d..62fea3c266a 100644 --- a/doc/user/admin_area/analytics/dev_ops_report.md +++ b/doc/user/admin_area/analytics/dev_ops_report.md @@ -4,21 +4,21 @@ group: Optimize info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# DevOps Report **(FREE SELF)** +# DevOps Reports **(FREE SELF)** -> [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/20976) from Conversational Development Index in GitLab 12.6. - -The DevOps Report gives you an overview of your entire instance's adoption of +DevOps Reports give you an overview of your entire instance's adoption of [Concurrent DevOps](https://about.gitlab.com/topics/concurrent-devops/) from planning to monitoring. -To see DevOps Report: +To see DevOps Reports: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Analytics > DevOps Report**. +1. On the left sidebar, select **Analytics > DevOps Reports**. ## DevOps Score +> [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/20976) from Conversational Development Index in GitLab 12.6. + NOTE: To see the DevOps score, you must activate your GitLab instance's [Service Ping](../settings/usage_statistics.md#service-ping). This is because DevOps Score is a comparative tool, so your score data must be centrally processed by GitLab Inc. first. @@ -72,4 +72,4 @@ DevOps Adoption allows you to: - Identify specific groups that are lagging in their adoption of GitLab, so you can help them along in their DevOps journey. - Find the groups that have adopted certain features, and can provide guidance to other groups on how to use those features. - + diff --git a/doc/user/admin_area/analytics/index.md b/doc/user/admin_area/analytics/index.md index dd1efc913fa..cd505e154c6 100644 --- a/doc/user/admin_area/analytics/index.md +++ b/doc/user/admin_area/analytics/index.md @@ -15,5 +15,5 @@ Administrators have access to instance-wide analytics: There are several kinds of statistics: -- [DevOps Report](dev_ops_report.md): Provides an overview of your entire instance's feature usage. +- [DevOps Reports](dev_ops_report.md): Provides an overview of your entire instance's feature usage. - [Usage Trends](usage_trends.md): Shows how much data your instance contains, and how that is changing. diff --git a/doc/user/admin_area/img/index_runners_search_or_filter_v14_1.png b/doc/user/admin_area/img/index_runners_search_or_filter_v14_1.png Binary files differdeleted file mode 100644 index ab196a0ca9e..00000000000 --- a/doc/user/admin_area/img/index_runners_search_or_filter_v14_1.png +++ /dev/null diff --git a/doc/user/admin_area/img/index_runners_search_or_filter_v14_5.png b/doc/user/admin_area/img/index_runners_search_or_filter_v14_5.png Binary files differnew file mode 100644 index 00000000000..10b8cc01103 --- /dev/null +++ b/doc/user/admin_area/img/index_runners_search_or_filter_v14_5.png diff --git a/doc/user/admin_area/index.md b/doc/user/admin_area/index.md index 27d2bd8f764..4de2397706b 100644 --- a/doc/user/admin_area/index.md +++ b/doc/user/admin_area/index.md @@ -107,6 +107,22 @@ You can combine the filter options. For example, to list only public projects wi 1. Click the **Public** tab. 1. Enter `score` in the **Filter by name...** input box. +#### Deleted projects **(PREMIUM SELF)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/37014) in GitLab 13.3. + +When delayed project deletion is [enabled for a group](../group/index.md#enable-delayed-project-removal), +projects within that group are not deleted immediately, but only after a delay. To access a list of all projects that are pending deletion: + +1. On the top bar, select **Menu > Projects > Explore projects**. +1. Select the **Deleted projects** tab. + +Listed for each project is: + +- The time the project was marked for deletion. +- The time the project is scheduled for final deletion. +- A **Restore** link to stop the project being eventually deleted. + ### Administering Users You can administer all users in the GitLab instance from the Admin Area's Users page: @@ -241,21 +257,27 @@ To [Create a new group](../group/index.md#create-a-group) click **New group**. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340920) in GitLab 14.4. -You can administer all topics in the GitLab instance from the Admin Area's Topics page. +You can administer all [topics](../project/working_with_projects.md#explore-topics) in the +GitLab instance from the Admin Area's Topics page. To access the Topics page: 1. On the top bar, select **Menu > Admin**. 1. On the left sidebar, select **Overview > Topics**. -For each topic, the page displays their name and number of projects labeled with the topic. +For each topic, the page displays its name and the number of projects labeled with the topic. To create a new topic, select **New topic**. To edit a topic, select **Edit** in that topic's row. To search for topics by name, enter your criteria in the search box. The topic search is case -insensitive, and applies partial matching. +insensitive and applies partial matching. + +NOTE: +The assigned topics are visible only to everyone with access to the project, +but everyone can see which topics exist at all on the GitLab instance. +Do not include sensitive information in the name of a topic. ### Administering Jobs @@ -309,11 +331,11 @@ To search runners' descriptions: You can also filter runners by status, type, and tag. To filter: -1. Click in the **Search or filter results...** field. -1. Select **Status**, **Type**, or **Tags**. +1. Select a tab or the **Search or filter results...** field. +1. Select any **Type**, or filter by **Status** or **Tags**. 1. Select or enter your search criteria. - + For each runner, the following attributes are listed: diff --git a/doc/user/admin_area/merge_requests_approvals.md b/doc/user/admin_area/merge_requests_approvals.md index 18151fc17d5..0ecf76902e1 100644 --- a/doc/user/admin_area/merge_requests_approvals.md +++ b/doc/user/admin_area/merge_requests_approvals.md @@ -5,34 +5,37 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, concepts --- -# Merge request approval rules **(PREMIUM SELF)** +# Merge request approvals **(PREMIUM SELF)** -> Introduced in [GitLab Premium](https://gitlab.com/gitlab-org/gitlab/-/issues/39060) 12.8. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39060) in GitLab 12.8. -Merge request approval rules prevent users from overriding certain settings on the project -level. When enabled at the instance level, these settings are no longer editable on the -project level. +Merge request approval rules prevent users from overriding certain settings on the project level. +When enabled at the instance level, these settings [cascade](../project/merge_requests/approvals/settings.md#settings-cascading) +and can no longer be changed: -To enable merge request approval rules for an instance: +- In projects. +- In groups. Cascading to groups was [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/285410) + in GitLab 14.5. + +To enable merge request approval settings for an instance: 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **{push-rules}** **Push Rules**, and expand **Merge request (MR) approvals**. -1. Set the required rule. +1. On the left sidebar, select **{push-rules}** **Push Rules**, and expand **Merge request approvals**. +1. Choose the required options. 1. Click **Save changes**. ## Available rules -Merge request approval rules that can be set at an instance level are: +Merge request approval settings that can be set at an instance level are: -- **Prevent approval by author**. Prevents project -maintainers from allowing request authors to merge their own merge requests. -- **Prevent approvals by users who add commits**. Prevents project -maintainers from allowing users to approve merge requests if they have submitted -any commits to the source branch. -- **Prevent editing approval rules in projects and merge requests**. Prevents users from -modifying the approvers list in project settings or in individual merge requests. +- **Prevent approval by author**. Prevents project maintainers from allowing request authors to + merge their own merge requests. +- **Prevent approvals by users who add commits**. Prevents project maintainers from allowing users + to approve merge requests if they have submitted any commits to the source branch. +- **Prevent editing approval rules in projects and merge requests**. Prevents users from modifying + the approvers list in project settings or in individual merge requests. See also the following, which are affected by instance-level rules: -- [Project-level merge request approval rules](../project/merge_requests/approvals/index.md). -- [Group-level merge request approval rules](../group/index.md#group-approval-rules) available in GitLab 13.9 and later. +- [Project merge request approval rules](../project/merge_requests/approvals/index.md). +- [Group merge request approval rules](../group/index.md#group-approval-rules) available in GitLab 13.9 and later. diff --git a/doc/user/admin_area/moderate_users.md b/doc/user/admin_area/moderate_users.md index c8f160f729f..a98250dfc56 100644 --- a/doc/user/admin_area/moderate_users.md +++ b/doc/user/admin_area/moderate_users.md @@ -17,7 +17,7 @@ pending approval state because an administrator has enabled any of the following - [Require admin approval for new sign-ups](settings/sign_up_restrictions.md#require-administrator-approval-for-new-sign-ups) setting. - [User cap](settings/sign_up_restrictions.md#user-cap). -- [Block auto-created users (OmniAuth)](../../integration/omniauth.md#initial-omniauth-configuration) +- [Block auto-created users (OmniAuth)](../../integration/omniauth.md#configure-initial-settings) - [Block auto-created users (LDAP)](../../administration/auth/ldap/index.md#basic-configuration-settings) When a user registers for an account while this setting is enabled: diff --git a/doc/user/admin_area/monitoring/background_migrations.md b/doc/user/admin_area/monitoring/background_migrations.md index da245300e1d..66001a987a4 100644 --- a/doc/user/admin_area/monitoring/background_migrations.md +++ b/doc/user/admin_area/monitoring/background_migrations.md @@ -24,7 +24,7 @@ prevent integer overflow for some tables. ## Check the status of background migrations All migrations must have a `Finished` status before you [upgrade GitLab](../../../update/index.md). -You can [check the status of existing migrations](../../../update/index.md#checking-for-background-migrations-before-upgrading). +You can [check the status of existing migrations](../../../update/index.md#batched-background-migrations). ## Enable or disable batched background migrations diff --git a/doc/user/admin_area/monitoring/health_check.md b/doc/user/admin_area/monitoring/health_check.md index c5ffb032afd..1d2d7be146c 100644 --- a/doc/user/admin_area/monitoring/health_check.md +++ b/doc/user/admin_area/monitoring/health_check.md @@ -1,8 +1,7 @@ --- -stage: none -group: unassigned +stage: Monitor +group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: concepts, howto --- # Health Check **(FREE SELF)** diff --git a/doc/user/admin_area/settings/account_and_limit_settings.md b/doc/user/admin_area/settings/account_and_limit_settings.md index 3549aa5323b..c511e85f3ce 100644 --- a/doc/user/admin_area/settings/account_and_limit_settings.md +++ b/doc/user/admin_area/settings/account_and_limit_settings.md @@ -61,10 +61,15 @@ details. ## Personal Access Token prefix +> [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/342327) in GitLab 14.5. Default prefix added. + You can set a global prefix for all generated Personal Access Tokens. A prefix can help you identify PATs visually, as well as with automation tools. +NOTE: +For GitLab.com and self-managed instances, the default prefix is `glpat-`. + ### Set a prefix Only a GitLab administrator can set the prefix, which is a global setting applied @@ -125,7 +130,7 @@ is rejected. NOTE: The repository size limit includes repository files and LFS, but does not include artifacts, uploads, -wiki, packages, or snippets. +wiki, packages, or snippets. The repository size limit applies to both private and public projects. For details on manually purging files, see [reducing the repository size using Git](../../project/repository/reducing_the_repo_size_using_git.md). @@ -148,7 +153,7 @@ add the line below to `/etc/gitlab/gitlab.rb` before increasing the max attachme nginx['client_max_body_size'] = "200m" ``` -## Customize session duration for Git Operations when 2FA is enabled **(PREMIUM)** +## Customize session duration for Git Operations when 2FA is enabled **(PREMIUM SELF)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/296669) in GitLab 13.9. > - It's deployed behind a feature flag, disabled by default. @@ -173,7 +178,7 @@ To set a limit on how long these sessions are valid: ## Limit the lifetime of personal access tokens **(ULTIMATE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3649) in GitLab Ultimate 12.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3649) in GitLab 12.6. Users can optionally specify a lifetime for [personal access tokens](../../profile/personal_access_tokens.md). @@ -222,7 +227,7 @@ Disabling SSH key expiration immediately enables all expired SSH keys. ## Allow expired Personal Access Tokens to be used **(ULTIMATE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214723) in GitLab Ultimate 13.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214723) in GitLab 13.1. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/296881) in GitLab 13.9. By default, expired personal access tokens (PATs) **are not usable**. diff --git a/doc/user/admin_area/settings/continuous_integration.md b/doc/user/admin_area/settings/continuous_integration.md index 1b6da0e2806..565e905d732 100644 --- a/doc/user/admin_area/settings/continuous_integration.md +++ b/doc/user/admin_area/settings/continuous_integration.md @@ -221,7 +221,7 @@ It is also possible to specify a [custom CI/CD configuration file for a specific By default, a banner displays in merge requests with no pipeline suggesting a walkthrough on how to add one. - + To enable or disable the banner: diff --git a/doc/user/admin_area/settings/deprecated_api_rate_limits.md b/doc/user/admin_area/settings/deprecated_api_rate_limits.md index a2edb6da86e..9be703f3b82 100644 --- a/doc/user/admin_area/settings/deprecated_api_rate_limits.md +++ b/doc/user/admin_area/settings/deprecated_api_rate_limits.md @@ -47,7 +47,7 @@ To override the general user and IP rate limits for requests to deprecated API e 1. Select the **Maximum authenticated API requests per period per user**. 1. Select the **Authenticated API rate limit period in seconds**. -## Resources +## Related topics - [Rate limits](../../../security/rate_limits.md) - [User and IP rate limits](user_and_ip_rate_limits.md) diff --git a/doc/user/admin_area/settings/files_api_rate_limits.md b/doc/user/admin_area/settings/files_api_rate_limits.md index f91d4b2fb0c..4f0f50dbcd2 100644 --- a/doc/user/admin_area/settings/files_api_rate_limits.md +++ b/doc/user/admin_area/settings/files_api_rate_limits.md @@ -49,7 +49,7 @@ To override the general user and IP rate limits for requests to the Repository f 1. Select the **Max authenticated API requests per period per user**. 1. Select the **Authenticated API rate limit period in seconds**. -## Resources +## Related topics - [Rate limits](../../../security/rate_limits.md) - [Repository files API](../../../api/repository_files.md) diff --git a/doc/user/admin_area/settings/git_lfs_rate_limits.md b/doc/user/admin_area/settings/git_lfs_rate_limits.md index 8a0754374e2..adc6cc2b11b 100644 --- a/doc/user/admin_area/settings/git_lfs_rate_limits.md +++ b/doc/user/admin_area/settings/git_lfs_rate_limits.md @@ -29,7 +29,7 @@ supersede the [general user and IP rate limits](user_and_ip_rate_limits.md): 1. Enter a value for **Authenticated Git LFS rate limit period in seconds**. 1. Select **Save changes**. -## Resources +## Related topics - [Rate limiting](../../../security/rate_limits.md) - [User and IP rate limits](user_and_ip_rate_limits.md) diff --git a/doc/user/admin_area/settings/img/suggest_pipeline_banner.png b/doc/user/admin_area/settings/img/suggest_pipeline_banner.png Binary files differdeleted file mode 100644 index 9f118ccc5ed..00000000000 --- a/doc/user/admin_area/settings/img/suggest_pipeline_banner.png +++ /dev/null diff --git a/doc/user/admin_area/settings/img/suggest_pipeline_banner_v14_5.png b/doc/user/admin_area/settings/img/suggest_pipeline_banner_v14_5.png Binary files differnew file mode 100644 index 00000000000..0d9bfa4a173 --- /dev/null +++ b/doc/user/admin_area/settings/img/suggest_pipeline_banner_v14_5.png diff --git a/doc/user/admin_area/settings/index.md b/doc/user/admin_area/settings/index.md index 688734849e5..7945e5d790f 100644 --- a/doc/user/admin_area/settings/index.md +++ b/doc/user/admin_area/settings/index.md @@ -7,124 +7,186 @@ type: index # Admin Area settings **(FREE SELF)** -As an administrator of a GitLab self-managed instance, you can manage the behavior of your deployment. To do so, select **Admin Area > Settings**. +As an administrator of a GitLab self-managed instance, you can manage the behavior of your +deployment. -The Admin Area is not accessible on GitLab.com, and settings can only be changed by the -GitLab.com administrators. See the [GitLab.com settings](../../gitlab_com/index.md) -documentation for all current settings and limits on the GitLab.com instance. +The **Admin Area** is not accessible on GitLab.com, and settings can only be changed by the +GitLab.com administrators. For the settings and limits on the GitLab.com instance, +read [GitLab.com settings](../../gitlab_com/index.md). -## General +## Access the Admin Area -To access the default page for Admin Area settings: +To access the **Admin Area**: +1. Sign in to your GitLab instance as an administrator. 1. On the top bar, select **Menu > Admin**. -1. On the left sidebar, select **Settings > General**. - -| Option | Description | -| ------ | ----------- | -| [Visibility and access controls](visibility_and_access_controls.md) | Set default and restrict visibility levels. Configure import sources and Git access protocol. | -| [Account and limit](account_and_limit_settings.md) | Set projects and maximum size limits, session duration, user options, and check feature availability for namespace plan. | -| [Diff limits](../diff_limits.md) | Diff content limits. | -| [Sign-up restrictions](sign_up_restrictions.md) | Configure the way a user creates a new account. | -| [Sign in restrictions](sign_in_restrictions.md) | Set requirements for a user to sign in. Enable mandatory two-factor authentication. | -| [Terms of Service and Privacy Policy](terms.md) | Include a Terms of Service agreement and Privacy Policy that all users must accept. | -| [External Authentication](external_authorization.md#configuration) | External Classification Policy Authorization | -| [Web terminal](../../../administration/integration/terminal.md#limiting-websocket-connection-time) | Set max session time for web terminal. | -| [Web IDE](../../project/web_ide/index.md#enable-live-preview) | Manage Web IDE features. | -| [FLoC](floc.md) | Enable or disable [Federated Learning of Cohorts (FLoC)](https://en.wikipedia.org/wiki/Federated_Learning_of_Cohorts) tracking. | - -## Integrations - -| Option | Description | -| ------ | ----------- | -| [Elasticsearch](../../../integration/elasticsearch.md#enable-advanced-search) | Elasticsearch integration. Elasticsearch AWS IAM. | -| [Kroki](../../../administration/integration/kroki.md#enable-kroki-in-gitlab) | Allow rendering of diagrams in AsciiDoc and Markdown documents using [kroki.io](https://kroki.io). | -| [Mailgun](../../../administration/integration/mailgun.md) | Enable your GitLab instance to receive invite email bounce events from Mailgun, if it is your email provider. | -| [PlantUML](../../../administration/integration/plantuml.md) | Allow rendering of PlantUML diagrams in documents. | -| [Slack application](../../../user/project/integrations/gitlab_slack_application.md#configuration) **(FREE SAAS)** | Slack integration allows you to interact with GitLab via slash commands in a chat window. This option is only available on GitLab.com, though it may be [available for self-managed instances in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/28164). | -| [Third party offers](third_party_offers.md) | Control the display of third party offers. | -| [Snowplow](../../../development/snowplow/index.md) | Configure the Snowplow integration. | -| [Google GKE](../../project/clusters/add_gke_clusters.md) | Google GKE integration allows you to provision GKE clusters from GitLab. | -| [Amazon EKS](../../project/clusters/add_eks_clusters.md) | Amazon EKS integration allows you to provision EKS clusters from GitLab. | - -## Repository - -| Option | Description | -| ------ | ----------- | -| [Repository's custom initial branch name](../../project/repository/branches/default.md#instance-level-custom-initial-branch-name) | Set a custom branch name for new repositories created in your instance. | -| [Repository mirror](visibility_and_access_controls.md#enable-project-mirroring) | Configure repository mirroring. | -| [Repository storage](../../../administration/repository_storage_types.md) | Configure storage path settings. | -| Repository maintenance | ([Repository checks](../../../administration/repository_checks.md) and [Housekeeping](../../../administration/housekeeping.md)). Configure automatic Git checks and housekeeping on repositories. | -| [Repository static objects](../../../administration/static_objects_external_storage.md) | Serve repository static objects (for example, archives and blobs) from an external storage (for example, a CDN). | - -## Templates **(PREMIUM SELF)** - -| Option | Description | -| ------ | ----------- | -| [Templates](instance_template_repository.md#configuration) | Set instance-wide template repository. | -| [Custom project templates](../custom_project_templates.md) | Select the custom project template source group. | - -## CI/CD - -| Option | Description | -| ------ | ----------- | -| [Continuous Integration and Deployment](continuous_integration.md) | Auto DevOps, runners and job artifacts. | -| [Required pipeline configuration](continuous_integration.md#required-pipeline-configuration) **(PREMIUM SELF)** | Set an instance-wide auto included [pipeline configuration](../../../ci/yaml/index.md). This pipeline configuration is run after the project's own configuration. | -| [Package Registry](continuous_integration.md#package-registry-configuration) | Settings related to the use and experience of using the GitLab Package Registry. Note there are [risks involved](../../packages/container_registry/index.md#use-with-external-container-registries) in enabling some of these settings. | - -## Reporting - -| Option | Description | -| ------ | ----------- | -| [Spam and Anti-bot Protection](../../../integration/recaptcha.md) | Enable reCAPTCHA or Akismet and set IP limits. For reCAPTCHA, we currently only support [v2](https://developers.google.com/recaptcha/docs/versions). | -| [Abuse reports](../review_abuse_reports.md) | Set notification email for abuse reports. | - -## Metrics and profiling - -| Option | Description | -| ------ | ----------- | -| [Metrics - Prometheus](../../../administration/monitoring/prometheus/gitlab_metrics.md) | Enable and configure Prometheus metrics. | -| [Metrics - Grafana](../../../administration/monitoring/performance/grafana_configuration.md#integration-with-gitlab-ui) | Enable and configure Grafana. | -| [Profiling - Performance bar](../../../administration/monitoring/performance/performance_bar.md#enable-the-performance-bar-for-non-administrators) | Enable access to the Performance Bar for non-administrator users in a given group. | -| [Self monitoring](../../../administration/monitoring/gitlab_self_monitoring_project/index.md#create-the-self-monitoring-project) | Enable or disable instance self monitoring. | -| [Usage statistics](usage_statistics.md) | Enable or disable version check and Service Ping. | -| [Pseudonymizer data collection](../../../administration/pseudonymizer.md) **(ULTIMATE)** | Enable or disable the Pseudonymizer data collection. | - -## Network - -| Option | Description | -| ------ | ----------- | -| Performance optimization | [Write to "authorized_keys" file](../../../administration/operations/fast_ssh_key_lookup.md#setting-up-fast-lookup-via-gitlab-shell) and [Push event activities limit and bulk push events](push_event_activities_limit.md). Various settings that affect GitLab performance. | -| [User and IP rate limits](user_and_ip_rate_limits.md) | Configure limits for web and API requests. | -| [Package Registry Rate Limits](package_registry_rate_limits.md) | Configure specific limits for Packages API requests that supersede the user and IP rate limits. | -| [Git LFS Rate Limits](git_lfs_rate_limits.md) | Configure specific limits for Git LFS requests that supersede the user and IP rate limits. | -| [Files API Rate Limits](files_api_rate_limits.md) | Configure specific limits for Files API requests that supersede the user and IP rate limits. | -| [Deprecated API Rate Limits](deprecated_api_rate_limits.md) | Configure specific limits for deprecated API requests that supersede the user and IP rate limits. | -| [Outbound requests](../../../security/webhooks.md) | Allow requests to the local network from hooks and services. | -| [Protected Paths](protected_paths.md) | Configure paths to be protected by Rack Attack. | -| [Incident Management](../../../operations/incident_management/index.md) Limits | Limit the number of inbound alerts that can be sent to a project. | -| [Notes creation limit](rate_limit_on_notes_creation.md)| Set a rate limit on the note creation requests. | - -## Geo - -| Option | Description | -| ------ | ----------- | -| Geo | Geo allows you to replicate your GitLab instance to other geographical locations. Redirects to **Admin Area > Geo > Settings** are no longer available at **Admin Area > Settings > Geo** in [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/36896). | - -## Preferences - -| Option | Description | -| ------ | ----------- | -| [Email](email.md) | Various email settings. | -| [What's new](../../../administration/whats-new.md) | Configure What's new drawer and content. | -| [Help page](help_page.md) | Help page text and support page URL. | -| [Pages](../../../administration/pages/index.md#custom-domain-verification) | Size and domain settings for static websites | -| [Polling interval multiplier](../../../administration/polling.md) | Configure how frequently the GitLab UI polls for updates. | -| [Gitaly timeouts](gitaly_timeouts.md) | Configure Gitaly timeouts. | -| Localization | [Default first day of the week](../../profile/preferences.md) and [Time tracking](../../project/time_tracking.md#limit-displayed-units-to-hours). | -| [Sidekiq Job Limits](sidekiq_job_limits.md) | Limit the size of Sidekiq jobs stored in Redis. | - -### Default first day of the week +1. On the left sidebar, select **Settings**, and the group of settings to view: + - [General](#general) + - [Geo](#geo) + - [CI/CD](#cicd) + - [Integrations](#integrations) + - [Metrics and profiling](#metrics-and-profiling) + - [Network](#network) + - [Preferences](#preferences) + - [Reporting](#reporting) + - [Repository](#repository) + - [Templates](#templates) + +### General + +The **General** settings contain: + +- [Visibility and access controls](visibility_and_access_controls.md) - Set default and + restrict visibility levels. Configure import sources and Git access protocol. +- [Account and limit](account_and_limit_settings.md) - Set projects and maximum size limits, + session duration, user options, and check feature availability for namespace plan. +- [Diff limits](../diff_limits.md) - Diff content limits. +- [Sign-up restrictions](sign_up_restrictions.md) - Configure the way a user creates a new account. +- [Sign in restrictions](sign_in_restrictions.md) - Set requirements for a user to sign in. + Enable mandatory two-factor authentication. +- [Terms of Service and Privacy Policy](terms.md) - Include a Terms of Service agreement + and Privacy Policy that all users must accept. +- [External Authentication](external_authorization.md#configuration) - External Classification Policy Authorization. +- [Web terminal](../../../administration/integration/terminal.md#limiting-websocket-connection-time) - + Set max session time for web terminal. +- [Web IDE](../../project/web_ide/index.md#enable-live-preview) - Manage Web IDE features. +- [FLoC](floc.md) - Enable or disable + [Federated Learning of Cohorts (FLoC)](https://en.wikipedia.org/wiki/Federated_Learning_of_Cohorts) tracking. + +### CI/CD + +The **CI/CD** settings contain: + +- [Continuous Integration and Deployment](continuous_integration.md) - + Auto DevOps, runners and job artifacts. +- [Required pipeline configuration](continuous_integration.md#required-pipeline-configuration) - + Set an instance-wide auto included [pipeline configuration](../../../ci/yaml/index.md). + This pipeline configuration is run after the project's own configuration. +- [Package Registry](continuous_integration.md#package-registry-configuration) - + Settings related to the use and experience of using the GitLab Package Registry. Some + [risks are involved](../../packages/container_registry/index.md#use-with-external-container-registries) + in enabling some of these settings. + +### Geo **(PREMIUM SELF)** + +The **Geo** setting contains: + +- [Geo](../../../administration/geo/index.md) - Replicate your GitLab instance to other + geographical locations. Redirects to **Admin Area > Geo > Settings** are no + longer available at **Admin Area > Settings > Geo** in [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/issues/36896). + +### Integrations + +The **Integrations** settings contain: + +- [Elasticsearch](../../../integration/elasticsearch.md#enable-advanced-search) - + Elasticsearch integration. Elasticsearch AWS IAM. +- [Kroki](../../../administration/integration/kroki.md#enable-kroki-in-gitlab) - + Allow rendering of diagrams in AsciiDoc and Markdown documents using [kroki.io](https://kroki.io). +- [Mailgun](../../../administration/integration/mailgun.md) - Enable your GitLab instance + to receive invite email bounce events from Mailgun, if it is your email provider. +- [PlantUML](../../../administration/integration/plantuml.md) - Allow rendering of PlantUML + diagrams in documents. +- [Slack application](../../../user/project/integrations/gitlab_slack_application.md#configuration) - + Slack integration allows you to interact with GitLab via slash commands in a chat window. + This option is only available on GitLab.com, though it may be + [available for self-managed instances in the future](https://gitlab.com/gitlab-org/gitlab/-/issues/28164). +- [Third party offers](third_party_offers.md) - Control the display of third-party offers. +- [Snowplow](../../../development/snowplow/index.md) - Configure the Snowplow integration. +- [Google GKE](../../project/clusters/add_gke_clusters.md) - Google GKE integration enables + you to provision GKE clusters from GitLab. +- [Amazon EKS](../../project/clusters/add_eks_clusters.md) - Amazon EKS integration enables + you to provision EKS clusters from GitLab. + +### Metrics and profiling + +The **Metrics and profiling** settings contain: + +- [Metrics - Prometheus](../../../administration/monitoring/prometheus/gitlab_metrics.md) - + Enable and configure Prometheus metrics. +- [Metrics - Grafana](../../../administration/monitoring/performance/grafana_configuration.md#integration-with-gitlab-ui) - + Enable and configure Grafana. +- [Profiling - Performance bar](../../../administration/monitoring/performance/performance_bar.md#enable-the-performance-bar-for-non-administrators) - + Enable access to the Performance Bar for non-administrator users in a given group. +- [Self monitoring](../../../administration/monitoring/gitlab_self_monitoring_project/index.md#create-the-self-monitoring-project) - + Enable or disable instance self monitoring. +- [Usage statistics](usage_statistics.md) - Enable or disable version check and Service Ping. +- [Pseudonymizer data collection](../../../administration/pseudonymizer.md) - + Enable or disable the Pseudonymizer data collection. + +### Network + +The **Network** settings contain: + +- Performance optimization - Various settings that affect GitLab performance, including: + - [Write to `authorized_keys` file](../../../administration/operations/fast_ssh_key_lookup.md#setting-up-fast-lookup-via-gitlab-shell). + - [Push event activities limit and bulk push events](push_event_activities_limit.md). +- [User and IP rate limits](user_and_ip_rate_limits.md) - Configure limits for web and API requests. + These rate limits can be overridden: + - [Package Registry Rate Limits](package_registry_rate_limits.md) - Configure specific + limits for Packages API requests that supersede the user and IP rate limits. + - [Git LFS Rate Limits](git_lfs_rate_limits.md) - Configure specific limits for + Git LFS requests that supersede the user and IP rate limits. + - [Files API Rate Limits](files_api_rate_limits.md) - Configure specific limits for + Files API requests that supersede the user and IP rate limits. + - [Deprecated API Rate Limits](deprecated_api_rate_limits.md) - Configure specific limits + for deprecated API requests that supersede the user and IP rate limits. +- [Outbound requests](../../../security/webhooks.md) - Allow requests to the local network from hooks and services. +- [Protected Paths](protected_paths.md) - Configure paths to be protected by Rack Attack. +- [Incident Management Limits](../../../operations/incident_management/index.md) - Limit the + number of inbound alerts that can be sent to a project. +- [Notes creation limit](rate_limit_on_notes_creation.md) - Set a rate limit on the note creation requests. + +### Preferences + +The **Preferences** settings contain: + +- [Email](email.md) - Various email settings. +- [What's new](../../../administration/whats-new.md) - Configure **What's new** drawer and content. +- [Help page](help_page.md) - Help page text and support page URL. +- [Pages](../../../administration/pages/index.md#custom-domain-verification) - + Size and domain settings for static websites. +- [Polling interval multiplier](../../../administration/polling.md) - + Configure how frequently the GitLab UI polls for updates. +- [Gitaly timeouts](gitaly_timeouts.md) - Configure Gitaly timeouts. +- Localization: + - [Default first day of the week](../../profile/preferences.md). + - [Time tracking](../../project/time_tracking.md#limit-displayed-units-to-hours). +- [Sidekiq Job Limits](sidekiq_job_limits.md) - Limit the size of Sidekiq jobs stored in Redis. + +### Reporting + +The **Reporting** settings contain: + +- [Spam and Anti-bot Protection](../../../integration/recaptcha.md) - + Enable anti-spam services, like reCAPTCHA or Akismet, and set IP limits. +- [Abuse reports](../review_abuse_reports.md) - Set notification email for abuse reports. + +### Repository + +The **Repository** settings contain: + +- [Repository's custom initial branch name](../../project/repository/branches/default.md#instance-level-custom-initial-branch-name) - + Set a custom branch name for new repositories created in your instance. +- [Repository mirror](visibility_and_access_controls.md#enable-project-mirroring) - + Configure repository mirroring. +- [Repository storage](../../../administration/repository_storage_types.md) - Configure storage path settings. +- Repository maintenance: + - [Repository checks](../../../administration/repository_checks.md) - Configure + automatic Git checks on repositories. + - [Housekeeping](../../../administration/housekeeping.md)). Configure automatic + Git housekeeping on repositories. +- [Repository static objects](../../../administration/static_objects_external_storage.md) - + Serve repository static objects (for example, archives and blobs) from an external storage (for example, a CDN). + +### Templates **(PREMIUM SELF)** + +The **Templates** settings contain: + +- [Templates](instance_template_repository.md#configuration) - Set instance-wide template repository. +- [Custom project templates](../custom_project_templates.md) - Select the custom project template source group. + +## Default first day of the week You can change the [Default first day of the week](../../profile/preferences.md) for the entire GitLab instance: diff --git a/doc/user/admin_area/settings/instance_template_repository.md b/doc/user/admin_area/settings/instance_template_repository.md index 044863729db..71eb7bbbdc9 100644 --- a/doc/user/admin_area/settings/instance_template_repository.md +++ b/doc/user/admin_area/settings/instance_template_repository.md @@ -7,7 +7,7 @@ type: reference # Instance template repository **(PREMIUM SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5986) in GitLab Premium 11.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5986) in GitLab 11.3. In hosted systems, enterprises often have a need to share their own templates across teams. This feature allows an administrator to pick a project to be the diff --git a/doc/user/admin_area/settings/protected_paths.md b/doc/user/admin_area/settings/protected_paths.md index dc328fe8b7c..e686c65fe9a 100644 --- a/doc/user/admin_area/settings/protected_paths.md +++ b/doc/user/admin_area/settings/protected_paths.md @@ -7,28 +7,11 @@ type: reference # Protected paths **(FREE SELF)** -Rate limiting is a common technique used to improve the security and durability -of a web application. For more details, see -[Rate limits](../../../security/rate_limits.md). +Rate limiting is a technique that improves the security and durability of a web +application. For more details, see [Rate limits](../../../security/rate_limits.md). -GitLab rate limits the following paths with Rack Attack by default: - -```plaintext -'/users/password', -'/users/sign_in', -'/api/#{API::API.version}/session.json', -'/api/#{API::API.version}/session', -'/users', -'/users/confirmation', -'/unsubscribes/', -'/import/github/personal_access_token', -'/admin/session' -``` - -GitLab responds with HTTP status code `429` to POST requests at protected paths -that exceed 10 requests per minute per IP address. - -See [User and IP rate limits](../../admin_area/settings/user_and_ip_rate_limits.md#response-headers) for the headers responded to blocked requests. +You can rate limit (protect) specified paths. For these paths, GitLab responds with HTTP status +code `429` to POST requests at protected paths that exceed 10 requests per minute per IP address. For example, the following are limited to a maximum 10 requests per minute: @@ -36,10 +19,15 @@ For example, the following are limited to a maximum 10 requests per minute: - User sign-up (if enabled) - User password reset -After 10 requests, the client must wait 60 seconds before it can -try again. +After 10 requests, the client must wait 60 seconds before it can try again. + +See also: + +- List of paths [protected by default](../../../administration/instance_limits.md#by-protected-path). +- [User and IP rate limits](../../admin_area/settings/user_and_ip_rate_limits.md#response-headers) + for the headers returned to blocked requests. -## Configure using GitLab UI +## Configure protected paths > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31246) in GitLab 12.4. diff --git a/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md b/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md index 020d02b1635..028d5e4c2f3 100644 --- a/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md +++ b/doc/user/admin_area/settings/rate_limits_on_raw_endpoints.md @@ -7,7 +7,7 @@ type: reference # Rate limits on raw endpoints **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30829) in GitLab 12.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/30635) in GitLab 12.2. This setting defaults to `300` requests per minute, and allows you to rate limit the requests to raw endpoints: @@ -21,7 +21,7 @@ For example, requests over `300` per minute to `https://gitlab.com/gitlab-org/gi This limit is: -- Applied independently per project, per commit and per file path. +- Applied independently per project, per file path. - Not applied per IP address. - Active by default. To disable, set the option to `0`. diff --git a/doc/user/admin_area/settings/sign_up_restrictions.md b/doc/user/admin_area/settings/sign_up_restrictions.md index ed80bca470e..8ce3b4f1c18 100644 --- a/doc/user/admin_area/settings/sign_up_restrictions.md +++ b/doc/user/admin_area/settings/sign_up_restrictions.md @@ -48,7 +48,7 @@ automatically approved in a background job. NOTE: This setting doesn't apply to LDAP or OmniAuth users. To enforce approvals for new users signing up using OmniAuth or LDAP, set `block_auto_created_users` to `true` in the -[OmniAuth configuration](../../../integration/omniauth.md#initial-omniauth-configuration) or +[OmniAuth configuration](../../../integration/omniauth.md#configure-initial-settings) or [LDAP configuration](../../../administration/auth/ldap/index.md#basic-configuration-settings). ## Require email confirmation diff --git a/doc/user/admin_area/settings/terms.md b/doc/user/admin_area/settings/terms.md index c7c41e665ec..693b3e6c7b6 100644 --- a/doc/user/admin_area/settings/terms.md +++ b/doc/user/admin_area/settings/terms.md @@ -28,13 +28,13 @@ To enforce acceptance of a Terms of Service and Privacy Policy: For each update to the terms, a new version is stored. When a user accepts or declines the terms, GitLab records which version they accepted or declined. +Existing users must accept the terms on their next GitLab interaction. +If a signed-in user declines the terms, they are signed out. + When enabled, it adds a mandatory checkbox to the sign up page for new users:  -Existing users must accept the terms on their next GitLab interaction. -If a logged-in user declines the terms, they are signed out. - <!-- ## Troubleshooting Include any troubleshooting steps that you can foresee. If you know beforehand what issues diff --git a/doc/user/admin_area/settings/user_and_ip_rate_limits.md b/doc/user/admin_area/settings/user_and_ip_rate_limits.md index ac6fe29da28..d713ef4b4e0 100644 --- a/doc/user/admin_area/settings/user_and_ip_rate_limits.md +++ b/doc/user/admin_area/settings/user_and_ip_rate_limits.md @@ -76,7 +76,9 @@ To enable the unauthenticated request rate limit: ## Use a custom rate limit response -A request that exceeds a rate limit returns a 429 response code and a +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/50693) in GitLab 13.8. + +A request that exceeds a rate limit returns a `429` response code and a plain-text body, which by default is `Retry later`. To use a custom response: @@ -88,7 +90,7 @@ To use a custom response: ## Response headers -> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/731) in GitLab 13.8, the `Rate-Limit` headers. `Retry-After` was introduced in an earlier version. +> [Introduced](https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/731) in GitLab 13.8, the `RateLimit` headers. `Retry-After` was introduced in an earlier version. When a client exceeds the associated rate limit, the following requests are blocked. The server may respond with rate-limiting information allowing the diff --git a/doc/user/admin_area/settings/visibility_and_access_controls.md b/doc/user/admin_area/settings/visibility_and_access_controls.md index 075becfd32f..fd44d6445cf 100644 --- a/doc/user/admin_area/settings/visibility_and_access_controls.md +++ b/doc/user/admin_area/settings/visibility_and_access_controls.md @@ -68,7 +68,7 @@ GitLab administrators can still update the default branch protection of a group. ## Define which roles can create projects Instance-level protections for project creation define which roles can -[add projects to a group](../../group/index.md#specify-who-can-add-projects-to-a-group)] +[add projects to a group](../../group/index.md#specify-who-can-add-projects-to-a-group) on the instance. To alter which roles have permission to create projects: 1. Sign in to GitLab as a user with [Administrator role](../../permissions.md). diff --git a/doc/user/analytics/ci_cd_analytics.md b/doc/user/analytics/ci_cd_analytics.md index 4a42133c308..e949f968c2b 100644 --- a/doc/user/analytics/ci_cd_analytics.md +++ b/doc/user/analytics/ci_cd_analytics.md @@ -21,6 +21,12 @@ View pipeline duration history:  +Pipeline statistics are gathered by collecting all available pipelines for the +project regardless of status. The data available for each individual day is based +on when the pipeline was created. The total pipeline calculation includes child +pipelines and pipelines that failed with invalid YAML. If you are interested in +filtering pipelines based on other attributes, consider using the [Pipelines API](../../api/pipelines.md#list-project-pipelines). + ## DevOps Research and Assessment (DORA) key metrics **(ULTIMATE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/275991) in GitLab 13.7. diff --git a/doc/user/analytics/code_review_analytics.md b/doc/user/analytics/code_review_analytics.md index fe091fc9899..496e768456f 100644 --- a/doc/user/analytics/code_review_analytics.md +++ b/doc/user/analytics/code_review_analytics.md @@ -22,7 +22,7 @@ Initially, no data appears. Data is populated as users comment on open merge req ## Overview -Code Review Analytics is available to users with Reporter access and above, and displays a table of open merge requests that have at least one non-author comment. The review time is measured from the time the first non-author comment was submitted. +Code Review Analytics is available to users with at least the Reporter [role](../permissions.md), and displays a table of open merge requests that have at least one non-author comment. The review time is measured from the time the first non-author comment was submitted. To access Code Review Analytics, from your project's menu, go to **Analytics > Code Review**. diff --git a/doc/user/analytics/img/product_analytics_commits_per_mr_v14_4.png b/doc/user/analytics/img/product_analytics_commits_per_mr_v14_4.png Binary files differnew file mode 100644 index 00000000000..649416c02c0 --- /dev/null +++ b/doc/user/analytics/img/product_analytics_commits_per_mr_v14_4.png diff --git a/doc/user/analytics/img/productivity_analytics_time_to_merge_v14_4.png b/doc/user/analytics/img/productivity_analytics_time_to_merge_v14_4.png Binary files differnew file mode 100644 index 00000000000..b5d0dd4d2ea --- /dev/null +++ b/doc/user/analytics/img/productivity_analytics_time_to_merge_v14_4.png diff --git a/doc/user/analytics/img/productivity_analytics_trendline_v14_4.png b/doc/user/analytics/img/productivity_analytics_trendline_v14_4.png Binary files differnew file mode 100644 index 00000000000..da5d3aec957 --- /dev/null +++ b/doc/user/analytics/img/productivity_analytics_trendline_v14_4.png diff --git a/doc/user/analytics/index.md b/doc/user/analytics/index.md index c01788f4fc4..0cc21e3f390 100644 --- a/doc/user/analytics/index.md +++ b/doc/user/analytics/index.md @@ -6,65 +6,6 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Analyze GitLab usage **(FREE)** -## Definitions - -When we describe GitLab analytics, we use the following terms: - -- **Cycle time:** The duration of only the execution work. Cycle time is often displayed in combination with the lead time, which is longer than the cycle time. GitLab measures cycle time from the earliest commit of a [linked issue's merge request](../project/issues/crosslinking_issues.md) to when that issue is closed. The cycle time approach underestimates the lead time because merge request creation is always later than commit time. GitLab displays cycle time in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](../analytics/value_stream_analytics.md). -- **Deploys:** The total number of successful deployments to production in the given time frame (across all applicable projects). GitLab displays deploys in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](value_stream_analytics.md). -- **DORA (DevOps Research and Assessment)** ["Four Keys"](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance): - - **Speed/Velocity** - - - **Deployment frequency:** The relative frequency of successful deployments to production - (hourly, daily, weekly, monthly, or yearly). - This effectively measures how often you are delivering value to end users. A higher deployment - frequency means you are able to get feedback and iterate more quickly in delivering - improvements and features faster. GitLab measures this as the number of deployments to a - [production environment](../../ci/environments/index.md#deployment-tier-of-environments) in - the given time period. - GitLab displays deployment frequency in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](value_stream_analytics.md). - - **Lead Time for Changes:** The time it takes for a commit to get into production. (1) GitLab - measures this as the median duration between merge request merge and deployment to a - [production environment](../../ci/environments/index.md#deployment-tier-of-environments) for - all MRs deployed in the given time period. This measure under-estimates lead time because - merge time is always later than commit time. The - [standard definition](https://github.com/GoogleCloudPlatform/fourkeys/blob/main/METRICS.md#lead-time-for-changes) uses median commit time. - [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/328459) to start - measuring from "issue first commit" as a better proxy, although still imperfect. - - - **Stability** - - **Change Failure Rate:** The percentage of deployments causing a failure in production. - GitLab measures this as the number of [incidents](../../operations/incident_management/incidents.md) - divided by the number of deployments to a - [production environment](../../ci/environments/index.md#deployment-tier-of-environments) in - the given time period. This assumes: - - - All incidents are related to a production environment. - - Incidents and deployments have a strictly one-to-one relationship (meaning any incident is - related to only one production deployment, and any production deployment is related to no - more than one incident). - - - **Time to Restore Service:** How long it takes an organization to recover from a failure in - production. GitLab measures this as the average time required to close the - [incidents](../../operations/incident_management/incidents.md) in the given time period. - This assumes: - - - All incidents are related to a [production environment](../../ci/environments/index.md#deployment-tier-of-environments). - - Incidents and deployments have a strictly one-to-one relationship (meaning any incident is related to only one production deployment, and any production deployment is related to no more than one incident). - -- **Lead time:** The duration of your value stream, from start to finish. Different from [Lead time for changes](#lead-time-for-changes). Often displayed in combination with "cycle time," which is shorter. GitLab measures lead time from issue creation to issue close. GitLab displays lead time in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md). -- **MTTC (Mean Time to Change):** The average duration between idea and delivery. GitLab measures MTTC from issue creation to the issue's latest related merge request's deployment to production. -- **MTTD (Mean Time to Detect):** The average duration that a bug goes undetected in production. GitLab measures MTTD from deployment of bug to issue creation. -- **MTTM (Mean Time To Merge):** The average lifespan of a merge request. GitLab measures MTTM from merge request creation to merge request merge (and closed/un-merged merge requests are excluded). For more information, see [Merge Request Analytics](merge_request_analytics.md). -- **MTTR (Mean Time to Recover/Repair/Resolution/Resolve/Restore):** The average duration that a bug is not fixed in production. GitLab measures MTTR from deployment of bug to deployment of fix. -- **Throughput:** The number of issues closed or merge requests merged (not closed) in some period of time. Often measured per sprint. GitLab displays merge request throughput in [Merge Request Analytics](merge_request_analytics.md). -- **Value Stream:** The entire work process that is followed to deliver value to customers. For example, the [DevOps lifecycle](https://about.gitlab.com/stages-devops-lifecycle/) is a value stream that starts with "plan" and ends with "monitor". GitLab helps you track your value stream using [Value Stream Analytics](value_stream_analytics.md). -- **Velocity:** The total issue burden completed in some period of time. The burden is usually measured in points or weight, often per sprint. For example, your velocity may be "30 points per sprint". GitLab measures velocity as the total points/weight of issues closed in a given period of time. - -### Lead time for changes - -"Lead Time for Changes" differs from ordinary "Lead Time" because it "focuses on measuring only the time to deliver a feature once it has been developed", as described in ([Measuring DevOps Performance](https://devops.com/measuring-devops-performance/)). - ## Instance-level analytics > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12077) in GitLab 12.2. @@ -112,3 +53,76 @@ The following analytics features are available for users to create personalized - [Application Security](../application_security/security_dashboard/#security-center) Be sure to review the documentation page for this feature for GitLab tier requirements. + +## Definitions + +We use the following terms to describe GitLab analytics: + +- **Cycle time:** The duration of only the execution work. Cycle time is often displayed in combination with the lead time, which is longer than the cycle time. GitLab measures cycle time from the earliest commit of a [linked issue's merge request](../project/issues/crosslinking_issues.md) to when that issue is closed. The cycle time approach underestimates the lead time because merge request creation is always later than commit time. GitLab displays cycle time in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](../analytics/value_stream_analytics.md). +- **Deploys:** The total number of successful deployments to production in the given time frame (across all applicable projects). GitLab displays deploys in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](value_stream_analytics.md). +- **DORA (DevOps Research and Assessment)** ["Four Keys"](https://cloud.google.com/blog/products/devops-sre/using-the-four-keys-to-measure-your-devops-performance): + - **Speed/Velocity** + + - **Deployment frequency:** The relative frequency of successful deployments to production + (hourly, daily, weekly, monthly, or yearly). + This measures how often you are delivering value to end users. A higher deployment + frequency means you are able to get feedback and iterate faster to deliver + improvements and features. GitLab measures this as the number of deployments to a + [production environment](../../ci/environments/index.md#deployment-tier-of-environments) in + the given time period. + GitLab displays deployment frequency in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md) and [project-level Value Stream Analytics](value_stream_analytics.md). + - **Lead Time for Changes:** The time it takes for a commit to get into production. GitLab + measures this as the median duration between merge request merge and deployment to a + [production environment](../../ci/environments/index.md#deployment-tier-of-environments) for + all MRs deployed in the given time period. This measure under estimates lead time because + merge time is always later than commit time. The + [standard definition](https://github.com/GoogleCloudPlatform/fourkeys/blob/main/METRICS.md#lead-time-for-changes) uses median commit time. + [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/328459) to start + measuring from "issue first commit" as a better proxy, although still imperfect. + + - **Stability** + - **Change Failure Rate:** The percentage of deployments causing a failure in production. + GitLab measures this as the number of [incidents](../../operations/incident_management/incidents.md) + divided by the number of deployments to a + [production environment](../../ci/environments/index.md#deployment-tier-of-environments) in + the given time period. This assumes: + + - All incidents are related to a production environment. + - Incidents and deployments have a strictly one-to-one relationship (meaning any incident is + related to only one production deployment, and any production deployment is related to no + more than one incident). + + - **Time to Restore Service:** How long it takes an organization to recover from a failure in + production. GitLab measures this as the average time required to close the + [incidents](../../operations/incident_management/incidents.md) in the given time period. + This assumes: + + - All incidents are related to a [production environment](../../ci/environments/index.md#deployment-tier-of-environments). + - Incidents and deployments have a strictly one-to-one relationship (meaning any incident is related to only one production deployment, and any production deployment is related to no more than one incident). + +- **Lead time:** The duration of your value stream, from start to finish. Different to +[Lead time for changes](#lead-time-for-changes). Often displayed in combination with "cycle time," +which is shorter. GitLab measures lead time from issue creation to issue close. GitLab displays lead +time in [group-level Value Stream Analytics](../group/value_stream_analytics/index.md). +- **Mean Time to Change (MTTC):** The average duration between idea and delivery. GitLab measures +MTTC from issue creation to the issue's latest related merge request's deployment to production. +- **Mean Time to Detect (MTTD):** The average duration that a bug goes undetected in production. +GitLab measures MTTD from deployment of bug to issue creation. +- **Mean Time To Merge (MTTM):** The average lifespan of a merge request. GitLab measures MTTM from +merge request creation to merge request merge (and closed/un-merged merge requests are excluded). +For more information, see [Merge Request Analytics](merge_request_analytics.md). +- **Mean Time to Recover/Repair/Resolution/Resolve/Restore (MTTR):** The average duration that a bug +is not fixed in production. GitLab measures MTTR from deployment of bug to deployment of fix. +- **Throughput:** The number of issues closed or merge requests merged (not closed) in a period of +time. Often measured per sprint. GitLab displays merge request throughput in [Merge Request Analytics](merge_request_analytics.md). +- **Value Stream:** The entire work process that is followed to deliver value to customers. For example, +the [DevOps lifecycle](https://about.gitlab.com/stages-devops-lifecycle/) is a value stream that starts +with "plan" and ends with "monitor". GitLab helps you track your value stream using [Value Stream Analytics](value_stream_analytics.md). +- **Velocity:** The total issue burden completed in some period of time. The burden is usually measured +in points or weight, often per sprint. For example, your velocity may be "30 points per sprint". GitLab +measures velocity as the total points or weight of issues closed in a given period of time. + +## Lead time for changes + +"Lead Time for Changes" differs from "Lead Time" because it "focuses on measuring only the time to +deliver a feature once it has been developed", as described in ([Measuring DevOps Performance](https://devops.com/measuring-devops-performance/)). diff --git a/doc/user/analytics/merge_request_analytics.md b/doc/user/analytics/merge_request_analytics.md index 2ff15c00a3f..bb110ab495f 100644 --- a/doc/user/analytics/merge_request_analytics.md +++ b/doc/user/analytics/merge_request_analytics.md @@ -116,4 +116,4 @@ bookmark for those preferred settings in your browser. The **Merge Request Analytics** feature can be accessed only: - On [GitLab Premium](https://about.gitlab.com/pricing/) and above. -- By users with [Reporter access](../permissions.md) and above. +- By users with at least the Reporter [role](../permissions.md). diff --git a/doc/user/analytics/productivity_analytics.md b/doc/user/analytics/productivity_analytics.md index dbe71d5f743..da55a0f093c 100644 --- a/doc/user/analytics/productivity_analytics.md +++ b/doc/user/analytics/productivity_analytics.md @@ -16,37 +16,79 @@ long, on average, it takes to deliver features is an enormous endeavor. While [Value Stream Analytics](../analytics/value_stream_analytics.md) focuses on the entire Software Development Life Cycle (SDLC) process, Productivity Analytics provides a way for Engineering Management to drill down in a systematic way to uncover patterns and causes for success or failure at an individual, project, or group level. -Productivity can slow down for many reasons ranging from degrading codebase to quickly growing teams. In order to investigate, department or team leaders can start by visualizing the time it takes for merge requests to be merged. +Productivity can slow down for many reasons ranging from degrading codebase to quickly growing teams. To investigate, department or team leaders can start by visualizing the time it takes for merge requests to be merged. -## Supported features +## Visualizations and metrics -Productivity Analytics allows GitLab users to: +With Productivity Analytics, GitLab users can: -- Visualize typical merge request (MR) lifetime and statistics. Use a histogram that shows the distribution of the time elapsed between creating and merging merge requests. -- Drill down into the most time consuming merge requests, select a number of outliers, and filter down all subsequent charts to investigate potential causes. +- Visualize typical merge request (MR) lifetime and statistics. A histogram shows the distribution of the time elapsed between creating and merging merge requests. +- Drill down into the most time consuming merge requests, select outliers, and filter subsequent charts to investigate potential causes. - Filter by group, project, author, label, milestone, or a specific date range. For example, filter down to the merge requests of a specific author in a group or project during a milestone or specific date range. -- Measure velocity over time. Visualize the trends of each metric from the charts above over time in order to observe progress. Zoom in on a particular date range if you notice outliers. +- Measure velocity over time. To observe progress, visualize the trends of each metric from the charts over time. Zoom in on a particular date range if you notice outliers. -## Accessing metrics and visualizations +## Metrics charts -To access the chart, navigate to a group's sidebar and select **Analytics > Productivity Analytics**. +To access the charts, navigate to a group's sidebar and select **Analytics > Productivity Analytics**. +Metrics and visualizations of **merged** merge requests are available on a project or group level. -The following metrics and visualizations are available on a project or group level - currently only covering **merged** merge requests: +### Time to merge -- Histogram showing the number of merge request that took a specified number of days to merge after creation. Select a specific column to filter down subsequent charts. -- Histogram showing a breakdown of the time taken (in hours) to merge a merge request. The following intervals are available: - - Time from first commit to first comment. - - Time from first comment until last commit. - - Time from last commit to merge. -- Histogram showing the size or complexity of a merge request, using the following: - - Number of commits per merge request. - - Number of lines of code per commit. - - Number of files touched. -- Scatterplot showing all MRs merged on a certain date, together with the days it took to complete the action and a 30 day rolling median. -- Table showing the list of merge requests with their respective time duration metrics. - - Users can sort by any of the above metrics. +The **Time to merge** histogram shows the number of merge requests and the number +of days it took to merge after creation. Select a column to filter subsequent charts. -## Date ranges + + +### Trendline + +The **Trendline** scatterplot shows all merge requests on a certain date, +and the days it took to complete the action and a 30 day rolling median. Select the dropdown to view: + +- Time from first commit to first comment. +- Time from first comment until last commit. +- Time from last commit to merge. +- Number of commits per merge request. +- Number of lines of code (LOC) per commit. +- Number of files touched. + + + +### Commits and merge request size + +Under the **Trendline** scatterplot, the left-side histogram shows +the time taken (in hours) between commits and comments until the merge +request is merged. Select the dropdown to view: + +- Time from first commit to first comment. +- Time from first comment until last commit. +- Time from last commit to merge. + +The right-side histogram shows the size or complexity of a merge request. +Select the dropdown to view: + +- Number of commits per merge request. +- Number of lines of code (LOC) per commit. +- Number of files touched. + + + +### Merge request list + +The **List** table shows a list of merge requests with their respective time duration metrics. + +Sort metrics by: + +- Time from first commit to first comment. +- Time from first comment until last commit. +- Time from last commit to merge. + +Filter metrics by: + +- Number of commits per merge request. +- Number of lines of code per commit. +- Number of files touched. + +## Filter by date range > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13188) in GitLab 12.4. @@ -61,4 +103,4 @@ You can filter analytics based on a date range. To filter results: The **Productivity Analytics** dashboard can be accessed only: - On [GitLab Premium](https://about.gitlab.com/pricing/) and above. -- By users with [Reporter access](../permissions.md) and above. +- By users with at least the Reporter [role](../permissions.md). diff --git a/doc/user/analytics/repository_analytics.md b/doc/user/analytics/repository_analytics.md index e8f8acf5661..936504187b4 100644 --- a/doc/user/analytics/repository_analytics.md +++ b/doc/user/analytics/repository_analytics.md @@ -29,7 +29,7 @@ Commits in a project's [wiki](../project/wiki/index.md#track-wiki-events) are no ### Charts -The data in the charts are updated soon after each commit in the default branch. +The data in the charts are queued. Background workers update the charts 10 minutes after each commit in the default branch. Depending on the size of the GitLab installation, it may take longer for data to refresh due to variations in the size of background job queues. Available charts: diff --git a/doc/user/application_security/api_fuzzing/index.md b/doc/user/application_security/api_fuzzing/index.md index e32989c2915..5cef0040ac3 100644 --- a/doc/user/application_security/api_fuzzing/index.md +++ b/doc/user/application_security/api_fuzzing/index.md @@ -111,12 +111,9 @@ To generate an API Fuzzing configuration snippet: ### OpenAPI Specification -> Support for OpenAPI Specification v3.1 was -> [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327268) in GitLab 14.2. -> Support for OpenAPI Specification using YAML format was -> [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. -> Support for OpenAPI Specification v3.0 was -> [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228652) in GitLab 13.9. +> - Support for OpenAPI Specification v3.0 was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/228652) in GitLab 13.9. +> - Support for OpenAPI Specification using YAML format was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/330583) in GitLab 14.0. +> - Support for OpenAPI Specification v3.1 was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327268) in GitLab 14.2. The [OpenAPI Specification](https://www.openapis.org/) (formerly the Swagger Specification) is an API description format for REST APIs. This section shows you how to configure API fuzzing using an OpenAPI Specification to provide information about the target API to test. @@ -214,7 +211,7 @@ To configure API fuzzing to use a HAR file: ``` 1. Provide the location of the HAR specification. You can provide the specification as a file - or URL. [URL support was introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285020) + or URL. URL support was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285020) in GitLab 13.10 and later. Specify the location by adding the `FUZZAPI_HAR` variable. 1. The target API instance's base URL is also required. Provide it by using the `FUZZAPI_TARGET_URL` @@ -285,7 +282,7 @@ To configure API fuzzing to use a Postman Collection file: ``` 1. Provide the location of the Postman Collection specification. You can provide the specification - as a file or URL. [URL support was introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285020) + as a file or URL. URL support was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285020) in GitLab 13.10 and later. Specify the location by adding the `FUZZAPI_POSTMAN_COLLECTION` variable. @@ -613,15 +610,15 @@ Overrides use a JSON document, where each type of override is represented by a J }, "body-form": { "form-param1": "value", - "form-param1": "value", + "form-param2": "value" }, "body-json": { "json-path1": "value", - "json-path2": "value", + "json-path2": "value" }, "body-xml" : { "xpath1": "value", - "xpath2": "value", + "xpath2": "value" } } ``` @@ -975,7 +972,7 @@ reported. ### View details of an API Fuzzing vulnerability -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.7. +> Introduced in GitLab 13.7. Faults detected by API Fuzzing occur in the live web application, and require manual investigation to determine if they are vulnerabilities. Fuzzing faults are included as vulnerabilities with a @@ -1156,12 +1153,41 @@ Profiles: ## Running API fuzzing in an offline environment -For self-managed GitLab instances in an environment with limited, restricted, or intermittent access -to external resources through the internet, some adjustments are required for the Web API Fuzz testing job to -successfully run. For more information, see [Offline environments](../offline_deployments/index.md). +For self-managed GitLab instances in an environment with limited, restricted, or intermittent access to external resources through the internet, some adjustments are required for the Web API Fuzz testing job to successfully run. + +Steps: + +1. Host the Docker image in a local container registry. +1. Set the `SECURE_ANALYZERS_PREFIX` to the local container registry. + +The Docker image for API Fuzzing must be pulled (downloaded) from the public registry and then pushed (imported) into a local registry. The GitLab container registry can be used to locally host the Docker image. This process can be performed using a special template. See [loading Docker images onto your offline host](../offline_deployments/index.md#loading-docker-images-onto-your-offline-host) for instructions. + +Once the Docker image is hosted locally, the `SECURE_ANALYZERS_PREFIX` variable is set with the location of the local registry. The variable must be set such that concatenating `/api-fuzzing:1` results in a valid image location. + +For example, the below line sets a registry for the image `registry.gitlab.com/gitlab-org/security-products/analyzers/api-fuzzing:1`: + +`SECURE_ANALYZERS_PREFIX: "registry.gitlab.com/gitlab-org/security-products/analyzers"` + +NOTE: +Setting `SECURE_ANALYZERS_PREFIX` changes the Docker image registry location for all GitLab Secure templates. + +For more information, see [Offline environments](../offline_deployments/index.md). ## Troubleshooting +### Error waiting for API Security 'http://127.0.0.1:5000' to become available + +A bug exists in versions of the API Fuzzing analyzer prior to v1.6.196 that can cause a background process to fail under certain conditions. The solution is to update to a newer version of the DAST API analyzer. + +The version information can be found in the job details for the `apifuzzer_fuzz` job. + +If the issue is occuring with versions v1.6.196 or greater, please contact Support and provide the following information: + +1. Reference this troubleshooting section and ask for the issue to be escalated to the Dynamic Analysis Team. +1. The full console output of the job. +1. The `gl-api-security-scanner.log` file available as a job artifact. In the right-hand panel of the job details page, select the **Browse** button. +1. The `apifuzzer_fuzz` job definition from your `.gitlab-ci.yml` file. + ### Error, the OpenAPI document is not valid. Errors were found during validation of the document using the published OpenAPI schema At the start of an API Fuzzing job the OpenAPI Specification is validated against the [published schema](https://github.com/OAI/OpenAPI-Specification/tree/master/schemas). This error is shown when the provided OpenAPI Specification has validation errors. Errors can be introduced when creating an OpenAPI Specification manually, and also when the schema is generated. diff --git a/doc/user/application_security/cluster_image_scanning/index.md b/doc/user/application_security/cluster_image_scanning/index.md index abbe00a85ab..790b428bac9 100644 --- a/doc/user/application_security/cluster_image_scanning/index.md +++ b/doc/user/application_security/cluster_image_scanning/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Cluster Image Scanning **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 14.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/) in GitLab 14.1. WARNING: This analyzer is in [Alpha](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha) @@ -26,10 +26,18 @@ GitLab provides integration with open-source tools for vulnerability analysis in To integrate GitLab with security scanners other than those listed here, see [Security scanner integration](../../../development/integrations/secure.md). -You can enable cluster image scanning by [including the CI job](#configuration) +You can use cluster image scanning through the following methods: + +- [The cluster image scanning analyzer](#use-the-cluster-image-scanning-analyzer) +- [The GitLab Kubernetes agent](#cluster-image-scanning-with-the-gitlab-kubernetes-agent) + +## Use the cluster image scanning analyzer + +You can use the cluster image scanning analyzer to run cluster image scanning with [GitLab CI/CD](../../../ci/index.md). +To enable the cluster image scanning analyzer, [include the CI job](#configuration) in your existing `.gitlab-ci.yml` file. -## Prerequisites +### Prerequisites To enable cluster image scanning in your pipeline, you need the following: @@ -45,7 +53,7 @@ To enable cluster image scanning in your pipeline, you need the following: [configuration variable](#cicd-variables-for-cluster-image-scanning) with the type set to `File` (see [Configuring the cluster](#configuring-the-cluster)). -## Configuring the cluster +### Configuring the cluster 1. Create a new service account. @@ -128,7 +136,7 @@ only. You can apply additional protection to your cluster by and [configuring Starboard Operator](https://aquasecurity.github.io/starboard/v0.10.3/operator/configuration/#install-modes) to install in restricted mode. -## Configuration +### Configuration To include the `Cluster-Image-Scanning.gitlab-ci.yml` template (GitLab 14.1 and later), add the following to your `.gitlab-ci.yml` file: @@ -149,7 +157,7 @@ GitLab saves the results as a that you can download and analyze later. When downloading, you always receive the most recent artifact. -### Customize the cluster image scanning settings +#### Customize the cluster image scanning settings You can customize how GitLab scans your cluster. For example, to restrict the analyzer to get results for only a certain workload, use the [`variables`](../../../ci/yaml/index.md#variables) @@ -157,7 +165,7 @@ parameter in your `.gitlab-ci.yml` to set [CI/CD variables](#cicd-variables-for- The variables you set in your `.gitlab-ci.yml` overwrite those in `Cluster-Image-Scanning.gitlab-ci.yml`. -#### CI/CD variables for cluster image scanning +##### CI/CD variables for cluster image scanning You can [configure](#customize-the-cluster-image-scanning-settings) analyzers by using the following CI/CD variables: @@ -168,8 +176,9 @@ You can [configure](#customize-the-cluster-image-scanning-settings) analyzers by | `CIS_RESOURCE_NAME` | `""` | Name of the Kubernetes resource you want to filter vulnerabilities for. For example, `nginx`. | | `CIS_RESOURCE_NAMESPACE` | `""` | Namespace of the Kubernetes resource you want to filter vulnerabilities for. For example, `production`. | | `CIS_RESOURCE_KIND` | `""` | Kind of the Kubernetes resource you want to filter vulnerabilities for. For example, `deployment`. | +| `CIS_CLUSTER_IDENTIFIER` | `""` | ID of the Kubernetes cluster integrated with GitLab. This is used to map vulnerabilities to the cluster so they can be filtered in the Vulnerability Report page. | -### Override the cluster image scanning template +#### Override the cluster image scanning template If you want to override the job definition (for example, to change properties like `variables`), you must declare and override a job after the template inclusion, and then @@ -186,7 +195,7 @@ cluster_image_scanning: CIS_RESOURCE_NAME: nginx ``` -### Connect with Kubernetes cluster associated to the project +#### Connect with Kubernetes cluster associated to the project If you want to connect to the Kubernetes cluster associated with the project and run Cluster Image Scanning jobs without configuring the `CIS_KUBECONFIG` variable, you must extend `cluster_image_scanning` and specify the environment you want to scan. @@ -200,7 +209,7 @@ cluster_image_scanning: action: prepare ``` -## Reports JSON format +### Reports JSON format The cluster image scanning tool emits a JSON report file. For more information, see the [schema for this report](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/container-scanning-report-format.json). @@ -208,7 +217,7 @@ The cluster image scanning tool emits a JSON report file. For more information, Here's an example cluster image scanning report: ```json-doc -{{ +{ "version": "14.0.2", "scan": { "scanner": { @@ -265,6 +274,27 @@ Here's an example cluster image scanning report: } ``` +## Cluster image scanning with the GitLab Kubernetes Agent + +You can use the [GitLab Kubernetes Agent](../../clusters/agent/index.md) to +scan images from within your Kubernetes cluster and record the vulnerabilities in GitLab. + +### Prerequisites + +- [Starboard Operator](https://aquasecurity.github.io/starboard/v0.10.3/operator/installation/kubectl/) + installed and configured in your cluster. +- [GitLab Kubernetes Agent](../../clusters/agent/install/index.md) + set up in GitLab, installed in your cluster, and configured using a configuration repository. + +### Configuration + +The GitLab Kubernetes agent begins to run cluster image scanning once the `cluster_image_scanning` +directive is added to your Kubernetes Agent configuration repository. + +See the [Kubernetes agent configuration repository](../../clusters/agent/repository.md#scan-your-container-images-for-vulnerabilities) +reference to learn more about the cluster image scanning configuration options for the +GitLab Kubernetes agent. + ## Security Dashboard The [Security Dashboard](../security_dashboard/index.md) shows you an overview of all diff --git a/doc/user/application_security/configuration/index.md b/doc/user/application_security/configuration/index.md index 674eee5e80a..a913d5fba92 100644 --- a/doc/user/application_security/configuration/index.md +++ b/doc/user/application_security/configuration/index.md @@ -7,9 +7,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Security Configuration **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20711) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.6. **(ULTIMATE)** -> - SAST configuration was [enabled](https://gitlab.com/groups/gitlab-org/-/epics/3659) in 13.3 and [improved](https://gitlab.com/gitlab-org/gitlab/-/issues/232862) in 13.4. **(ULTIMATE)** -> - DAST Profiles feature was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40474) in 13.4. **(ULTIMATE)** +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/20711) in GitLab 12.6. +> - SAST configuration was [enabled](https://gitlab.com/groups/gitlab-org/-/epics/3659) in 13.3 and [improved](https://gitlab.com/gitlab-org/gitlab/-/issues/232862) in 13.4. +> - DAST Profiles feature was [introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40474) in 13.4. > - A simplified version was made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/294076) in GitLab 13.10. > - [Redesigned](https://gitlab.com/gitlab-org/gitlab/-/issues/326926) in 14.2. @@ -38,31 +38,31 @@ Select **Configuration history** to see the `.gitlab-ci.yml` file's history. You can configure the following security controls: -- Static Application Security Testing (SAST) **(FREE)** +- [Static Application Security Testing](../sast/index.md) (SAST) - Select **Enable SAST** to configure SAST for the current project. For more details, read [Configure SAST in the UI](../sast/index.md#configure-sast-in-the-ui). -- Dynamic Application Security Testing (DAST) **(ULTIMATE)** +- [Dynamic Application Security Testing](../dast/index.md) (DAST) - Select **Enable DAST** to configure DAST for the current project. - Select **Manage scans** to manage the saved DAST scans, site profiles, and scanner profiles. For more details, read [DAST on-demand scans](../dast/index.md#on-demand-scans). -- Dependency Scanning **(ULTIMATE)** +- [Dependency Scanning](../dependency_scanning/index.md) - Select **Configure via Merge Request** to create a merge request with the changes required to enable Dependency Scanning. For more details, see [Enable Dependency Scanning via an automatic merge request](../dependency_scanning/index.md#enable-dependency-scanning-via-an-automatic-merge-request). -- Container Scanning **(ULTIMATE)** +- [Container Scanning](../container_scanning/index.md) - Can be configured with `.gitlab-ci.yml`. For more details, read [Container Scanning](../../../user/application_security/container_scanning/index.md#configuration). -- Cluster Image Scanning **(ULTIMATE)** +- [Cluster Image Scanning](../cluster_image_scanning/index.md) - Can be configured with `.gitlab-ci.yml`. For more details, read [Cluster Image Scanning](../../../user/application_security/cluster_image_scanning/#configuration). -- Secret Detection +- [Secret Detection](../secret_detection/index.md) - Select **Configure via Merge Request** to create a merge request with the changes required to enable Secret Detection. For more details, read [Enable Secret Detection via an automatic merge request](../secret_detection/index.md#enable-secret-detection-via-an-automatic-merge-request). -- API Fuzzing **(ULTIMATE)** +- [API Fuzzing](../api_fuzzing/index.md) - Select **Enable API Fuzzing** to use API Fuzzing for the current project. For more details, read [API Fuzzing](../../../user/application_security/api_fuzzing/index.md#enable-web-api-fuzzing). -- Coverage Fuzzing **(ULTIMATE)** +- [Coverage Fuzzing](../coverage_fuzzing/index.md) - Can be configured with `.gitlab-ci.yml`. For more details, read [Coverage Fuzzing](../../../user/application_security/coverage_fuzzing/index.md#configuration). ## Compliance **(ULTIMATE)** You can configure the following security controls: -- License Compliance **(ULTIMATE)** +- [License Compliance](../../../user/compliance/license_compliance/index.md) - Can be configured with `.gitlab-ci.yml`. For more details, read [License Compliance](../../../user/compliance/license_compliance/index.md#configuration). diff --git a/doc/user/application_security/container_scanning/index.md b/doc/user/application_security/container_scanning/index.md index 22b54bf019c..da2816ab6ed 100644 --- a/doc/user/application_security/container_scanning/index.md +++ b/doc/user/application_security/container_scanning/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Container Scanning **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3672) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/3672) in GitLab 10.4. Your application's Docker image may itself be based on Docker images that contain known vulnerabilities. By including an extra job in your pipeline that scans for those vulnerabilities and @@ -148,7 +148,7 @@ Support depends on the scanner: #### UBI-based images -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5775) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 14.1. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5775) in GitLab 14.1. GitLab also offers [Red Hat UBI](https://www.redhat.com/en/blog/introducing-red-hat-universal-base-image) versions of the container-scanning images. You can therefore replace standard images with UBI-based diff --git a/doc/user/application_security/coverage_fuzzing/index.md b/doc/user/application_security/coverage_fuzzing/index.md index cdb2e7109bf..9bb13d26d90 100644 --- a/doc/user/application_security/coverage_fuzzing/index.md +++ b/doc/user/application_security/coverage_fuzzing/index.md @@ -70,7 +70,7 @@ my_fuzz_target: - ./gitlab-cov-fuzz run --regression=$REGRESSION -- <your fuzz target> ``` -The included template makes available the [hidden job](../../../ci/yaml/index.md#hide-jobs) +The included template makes available the [hidden job](../../../ci/jobs/index.md#hide-jobs) `.fuzz_base`, which you must [extend](../../../ci/yaml/index.md#extends) for each of your fuzz targets. Each fuzz target **must** have a separate job. For example, the [go-fuzzing-example project](https://gitlab.com/gitlab-org/security-products/demos/go-fuzzing-example) @@ -146,7 +146,7 @@ corpus. ### Reports JSON format -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220062) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.3 as an [Alpha feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220062) in GitLab 13.3 as an [Alpha feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha). The `gitlab-cov-fuzz` tool emits a JSON report file. For more information, see the [schema for this report](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/coverage-fuzzing-report-format.json). diff --git a/doc/user/application_security/dast/browser_based.md b/doc/user/application_security/dast/browser_based.md index 9c5b84f4f36..10ca3430b48 100644 --- a/doc/user/application_security/dast/browser_based.md +++ b/doc/user/application_security/dast/browser_based.md @@ -54,6 +54,7 @@ The browser-based crawler can be configured using CI/CD variables. | `DAST_BROWSER_SCAN` | boolean | `true` | Configures DAST to use the browser-based crawler engine. | | `DAST_BROWSER_ALLOWED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered in scope when crawled. By default the `DAST_WEBSITE` hostname is included in the allowed hosts list. | | `DAST_BROWSER_EXCLUDED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are considered excluded and connections are forcibly dropped. | +| `DAST_BROWSER_EXCLUDED_ELEMENTS` | selector | `a[href='2.html'],css:.no-follow` | Comma-separated list of selectors that are ignored when scanning. | | `DAST_BROWSER_IGNORED_HOSTS` | List of strings | `site.com,another.com` | Hostnames included in this variable are accessed but not reported against. | | `DAST_BROWSER_MAX_ACTIONS` | number | `10000` | The maximum number of actions that the crawler performs. For example, clicking a link, or filling a form. | | `DAST_BROWSER_MAX_DEPTH` | number | `10` | The maximum number of chained actions that the crawler takes. For example, `Click -> Form Fill -> Click` is a depth of three. | @@ -127,7 +128,6 @@ dast: DAST_BROWSER_ACTION_TIMEOUT: "10s" DAST_BROWSER_STABILITY_TIMEOUT: "15s" DAST_BROWSER_NAVIGATION_STABILITY_TIMEOUT: "15s" - DAST_BROWSER_ACTION_TIMEOUT: "10s" DAST_BROWSER_ACTION_STABILITY_TIMEOUT: "3s" ``` diff --git a/doc/user/application_security/dast/checks/1004.1.md b/doc/user/application_security/dast/checks/1004.1.md new file mode 100644 index 00000000000..cbbcea1d34d --- /dev/null +++ b/doc/user/application_security/dast/checks/1004.1.md @@ -0,0 +1,41 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sensitive cookie without `HttpOnly` attribute + +## Description + +The {cookie_name} cookie was transmitted in a `Set-Cookie` header without the `HttpOnly` attribute set. +To prevent JavaScript being able to access the cookie value - usually via `document.cookies` - all +cookies that are used for authorization or contain sensitive information should have the `HttpOnly` attribute +set. + +## Remediation + +Most web application frameworks allow configuring how cookies are sent to user-agents. Consult your framework's +documentation for more information on how to enable various security directives when assigning cookies to clients. + +If the application is assigning cookies via writing to the response headers directly, ensure all responses include +the `HttpOnly` attribute. By enabling this protection, the application is able to mitigate the impact of +certain Cross-Site Scripting (XSS) attacks. + +Example: + +```http +Set-Cookie: {cookie_name}=<random secure value>; HttpOnly +``` + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 1004.1 | false | 1004 | Passive | Low | + +## Links + +- [owasp](https://owasp.org/www-community/HttpOnly) +- [cwe](https://cwe.mitre.org/data/definitions/1004.html) +- [Mozilla MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies) diff --git a/doc/user/application_security/dast/checks/16.1.md b/doc/user/application_security/dast/checks/16.1.md new file mode 100644 index 00000000000..bb030d2f9c4 --- /dev/null +++ b/doc/user/application_security/dast/checks/16.1.md @@ -0,0 +1,33 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Missing Content-Type header + +## Description + +The `Content-Type` header ensures that user agents correctly interpret the data being received. Without this header +being sent, the browser may misinterpret the data, leading to MIME confusion attacks. If an attacker were able +to upload files that are accessible by using a browser, they could upload files that may be interpreted as +HTML and so execute Cross-Site Scripting (XSS) attacks. + +## Remediation + +Ensure all resources return a proper `Content-Type` header that matches their format. As an example, +when returning JavaScript files, the response header should be: `Content-Type: application/javascript` + +For added protection, we recommend that all resources return the `X-Content-Type-Options: nosniff` +header to disable user agents from mis-interpreting resources. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 16.1 | true | 16 | Passive | Low | + +## Links + +- [cwe](https://cwe.mitre.org/data/definitions/16.html) +- [Mozilla Blog on MIME Confusion attacks](https://blog.mozilla.org/security/2016/08/26/mitigating-mime-confusion-attacks-in-firefox/) diff --git a/doc/user/application_security/dast/checks/16.2.md b/doc/user/application_security/dast/checks/16.2.md new file mode 100644 index 00000000000..95461e8677d --- /dev/null +++ b/doc/user/application_security/dast/checks/16.2.md @@ -0,0 +1,44 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Server header exposes version information + +## Description + +The target website returns the `Server` header and version information of this website. By +exposing these values, attackers may attempt to identify if the target software is vulnerable to known +vulnerabilities, or catalog known sites running particular versions to exploit in the future when a +vulnerability is identified in the particular version. + +## Remediation + +We recommend that the version information be removed from the `Server` header. + +Apache: +For Apache based web sites, set the `ServerTokens` to `Prod` in the `httpd.conf` configuration file. + +NGINX: +For NGINX based websites, set the `server_tokens` configuration value to `off` in the `nginx.conf` file. + +IIS: +For IIS based websites version 10 and above you can use the `removeServerHeader` element to the `requestFiltering` +section of the `Web.config` file. + +For all other server types, please consult your product's documentation on how to redact the version information from +the `Server` header. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 16.2 | true | 16 | Passive | Low | + +## Links + +- [cwe](https://cwe.mitre.org/data/definitions/16.html) +- [Apache ServerTokens](https://blog.mozilla.org/security/2016/08/26/mitigating-mime-confusion-attacks-in-firefox/) +- [NGINX server_tokens](https://nginx.org/en/docs/http/ngx_http_core_module.html#server_tokens) +- [IIS 10 Remove Server Header](https://docs.microsoft.com/en-us/iis/configuration/system.webserver/security/requestfiltering/#attributes) diff --git a/doc/user/application_security/dast/checks/16.3.md b/doc/user/application_security/dast/checks/16.3.md new file mode 100644 index 00000000000..e4dcf3ece4b --- /dev/null +++ b/doc/user/application_security/dast/checks/16.3.md @@ -0,0 +1,35 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# X-Powered-By header exposes version information + +## Description + +The target website returns the `X-Powered-By` header and version information of this website. By +exposing these values, attackers may attempt to identify if the target software is vulnerable to known +vulnerabilities, or catalog known sites running particular versions to exploit in the future when a +vulnerability is identified in the particular version. + +## Remediation + +We recommend that the version information be removed from the `X-Powered-By` header. + +PHP: +For PHP based web sites, set the `expose_php` option to `off` in the `php.ini` configuration file. + +For all other server types, please consult your product's documentation on how to redact the version +information from the `X-Powered-By` header. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 16.3 | true | 16 | Passive | Low | + +## Links + +- [cwe](https://cwe.mitre.org/data/definitions/16.html) +- [PHP expose_php](https://www.php.net/manual/en/ini.core.php#ini.expose-php) diff --git a/doc/user/application_security/dast/checks/16.4.md b/doc/user/application_security/dast/checks/16.4.md new file mode 100644 index 00000000000..c0161c910b0 --- /dev/null +++ b/doc/user/application_security/dast/checks/16.4.md @@ -0,0 +1,28 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# X-Backend-Server header exposes server information + +## Description + +The target website returns the `X-Backend-Server` header which includes potentially internal/hidden IP addresses +or hostnames. By exposing these values, attackers may attempt to circumvent security proxies and access these +hosts directly. + +## Remediation + +Consult your proxy/load balancer documentation or provider on how to disable revealing the +`X-Backend-Server` header value. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 16.4 | true | 16 | Passive | Info | + +## Links + +- [cwe](https://cwe.mitre.org/data/definitions/16.html) diff --git a/doc/user/application_security/dast/checks/16.5.md b/doc/user/application_security/dast/checks/16.5.md new file mode 100644 index 00000000000..8a6f3cd8b6a --- /dev/null +++ b/doc/user/application_security/dast/checks/16.5.md @@ -0,0 +1,30 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# AspNet Header(s) exposes version information + +## Description + +The target website returns AspNet header(s) and version information of this website. By +exposing these values attackers may attempt to identify if the target software is vulnerable to known +vulnerabilities, or catalog known sites running particular versions to exploit in the future when a +vulnerability is identified in the particular version. + +## Remediation + +To remove the `X-AspNet-Version` header set `<httpRuntime enableVersionHeader="false" />` in the `<system.Web>` +section of the `Web.config` file. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 16.5 | true | 16 | Passive | Low | + +## Links + +- [cwe](https://cwe.mitre.org/data/definitions/16.html) +- [IIS Remove Unwanted Headers](https://techcommunity.microsoft.com/t5/iis-support-blog/remove-unwanted-http-response-headers/ba-p/369710) diff --git a/doc/user/application_security/dast/checks/614.1.md b/doc/user/application_security/dast/checks/614.1.md new file mode 100644 index 00000000000..74ac73935f1 --- /dev/null +++ b/doc/user/application_security/dast/checks/614.1.md @@ -0,0 +1,40 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Sensitive cookie without `Secure` attribute + +## Description + +The {cookie_name} cookie was transmitted in a `Set-Cookie` response without the `Secure` attribute set. +To prevent sensitive cookie values being accidentally transmitted over clear-text HTTP we +recommended that cookies are declared with the `Secure` attribute. + +## Remediation + +Most web application frameworks allow configuring how cookies are sent to user agents. Consult your framework's +documentation for more information on how to enable various security attributes when assigning cookies to clients. + +If the application is assigning cookies via writing to the response headers directly, ensure all responses include +the `Secure` attribute. By enabling this protection, the application will no longer send sensitive cookies over +HTTP. + +Example: + +```http +Set-Cookie: {cookie_name}=<random secure value>; Secure +``` + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 614.1 | false | 614 | Passive | Low | + +## Links + +- [owasp](https://owasp.org/www-community/controls/SecureCookieAttribute) +- [cwe](https://cwe.mitre.org/data/definitions/614.html) +- [Mozilla MDN](https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies#restrict_access_to_cookies) diff --git a/doc/user/application_security/dast/checks/693.1.md b/doc/user/application_security/dast/checks/693.1.md new file mode 100644 index 00000000000..07cb368b39a --- /dev/null +++ b/doc/user/application_security/dast/checks/693.1.md @@ -0,0 +1,36 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Missing X-Content-Type-Options: nosniff + +## Description + +The `X-Content-Type-Options` header with the value `nosniff` ensures that user agents do not attempt to +guess the format of the data being received. User Agents such as browsers, commonly attempt to guess +what the resource type being requested is, through a process called MIME type sniffing. + +Without this header being sent, the browser may misinterpret the data, leading to MIME confusion attacks. +If an attacker were able to upload files that are accessible by using a browser, they could upload files +that could be interpreted as HTML and execute Cross-Site Scripting (XSS) attacks. + +## Remediation + +We recommend that the header and value of `X-Content-Type-Options: nosniff` be set server wide. +This ensures any resources that are mistakenly missing a `Content-Type` value are not +misinterpreted. + +## Details + +| ID | Aggregated | CWE | Type | Risk | +|:---|:--------|:--------|:--------|:--------| +| 693.1 | true | 693 | Passive | Low | + +## Links + +- [owasp](https://owasp.org/www-project-secure-headers/#x-content-type-options) +- [cwe](https://cwe.mitre.org/data/definitions/693.html) +- [Mozilla Blog on MIME Confusion attacks](https://blog.mozilla.org/security/2016/08/26/mitigating-mime-confusion-attacks-in-firefox/) +- [Mozilla MDN on X-Content-Type-Options](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-Content-Type-Options) diff --git a/doc/user/application_security/dast/checks/index.md b/doc/user/application_security/dast/checks/index.md new file mode 100644 index 00000000000..f1a68387eb1 --- /dev/null +++ b/doc/user/application_security/dast/checks/index.md @@ -0,0 +1,20 @@ +--- +stage: Secure +group: Dynamic Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# DAST browser-based crawler vulnerability checks **(ULTIMATE)** + +The [DAST browser-based crawler](../browser_based.md) provides a number of vulnerability checks that are used to scan for vulnerabilities in the site under test. + +| ID | Check | Severity | Type | +|:---|:------|:---------|:-----| +| [1004.1](1004.1.md) | Sensitive cookie without `HttpOnly` attribute | Low | Passive | +| [16.1](16.1.md) | Missing Content-Type header | Low | Passive | +| [16.2](16.2.md) | Server header exposes version information | Low | Passive | +| [16.3](16.3.md) | X-Powered-By header exposes version information | Low | Passive | +| [16.4](16.4.md) | X-Backend-Server header exposes server information | Info | Passive | +| [16.5](16.5.md) | AspNet Header(s) exposes version information | Low | Passive | +| [614.1](614.1.md) | Sensitive cookie without `Secure` attribute | Low | Passive | +| [693.1](693.1.md) | Missing X-Content-Type-Options: nosniff | Low | Passive | diff --git a/doc/user/application_security/dast/index.md b/doc/user/application_security/dast/index.md index 09b55e7b395..0d8b55a92a9 100644 --- a/doc/user/application_security/dast/index.md +++ b/doc/user/application_security/dast/index.md @@ -320,8 +320,8 @@ tips for optimizing DAST scans in a [blog post](https://about.gitlab.com/blog/20 ### API scan -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10928) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.10. -> - A new DAST API scanning engine was introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10928) in GitLab 12.10. +> - A new DAST API scanning engine was introduced in GitLab 13.10. Using an API specification as a scan's target is a useful way to seed URLs for scanning an API. Vulnerability rules in an API scan are different than those in a normal website scan. @@ -416,7 +416,7 @@ variables: ### URL scan -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.4. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214120) in GitLab 13.4. > - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/273141) in GitLab 13.11. A URL scan allows you to specify which parts of a website are scanned by DAST. @@ -492,7 +492,7 @@ Click **View details** to view the web console output which includes the list of ### View details of a vulnerability detected by DAST -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.1. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36332) in GitLab 13.1. Vulnerabilities detected by DAST occur in the live web application. Addressing these types of vulnerabilities requires specific information. DAST provides the information required to @@ -954,6 +954,11 @@ An on-demand scan can be run in active or passive mode: minimize the risk of accidental damage, running an active scan requires a [validated site profile](#site-profile-validation). +### View on-demand DAST scans + +To view running and completed on-demand DAST scans for a project, go to +**Security & Compliance > On-demand Scans** in the left sidebar. + ### Run an on-demand DAST scan Prerequisites: @@ -987,6 +992,7 @@ To run an on-demand scan either at a scheduled date or frequency, read 1. From your project's home page, go to **Security & Compliance > On-demand Scans** in the left sidebar. +1. Select **New DAST scan**. 1. Complete the **Scan name** and **Description** fields. 1. In GitLab 13.10 and later, select the desired branch from the **Branch** dropdown. 1. In **Scanner profile**, select a scanner profile from the dropdown. @@ -1017,17 +1023,13 @@ The on-demand DAST scan runs, and the project's dashboard shows the results. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.3. [Deployed behind the `dast_on_demand_scans_scheduler` flag](../../../administration/feature_flags.md), disabled by default. > - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.4. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, ask an -administrator to [disable the feature flag](../../../administration/feature_flags.md) named -`dast_on_demand_scans_scheduler`. -On GitLab.com, this feature is available. +> - [Feature flag dast_on_demand_scans_scheduler removed](https://gitlab.com/gitlab-org/gitlab/-/issues/328749) in GitLab 14.5. To schedule a scan: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Security & Compliance > On-demand Scans**. +1. Select **New DAST scan**. 1. Complete the **Scan name** and **Description** text boxes. 1. In GitLab 13.10 and later, from the **Branch** dropdown list, select the desired branch. 1. In the **Scanner profile** section, from the dropdown list, select a scanner profile. diff --git a/doc/user/application_security/dast/run_dast_offline.md b/doc/user/application_security/dast/run_dast_offline.md index 39747a5cbe5..86621d73524 100644 --- a/doc/user/application_security/dast/run_dast_offline.md +++ b/doc/user/application_security/dast/run_dast_offline.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto --- -# Run DAST in an offline environment +# Run DAST in an offline environment **(ULTIMATE)** For self-managed GitLab instances in an environment with limited, restricted, or intermittent access to external resources through the internet, some adjustments are required for the DAST job to diff --git a/doc/user/application_security/dast_api/index.md b/doc/user/application_security/dast_api/index.md index 3b1c91b0be4..f3ab25ccffa 100644 --- a/doc/user/application_security/dast_api/index.md +++ b/doc/user/application_security/dast_api/index.md @@ -681,15 +681,15 @@ Overrides use a JSON document, where each type of override is represented by a J }, "body-form": { "form-param1": "value", - "form-param1": "value", + "form-param2": "value" }, "body-json": { "json-path1": "value", - "json-path2": "value", + "json-path2": "value" }, "body-xml" : { "xpath1": "value", - "xpath2": "value", + "xpath2": "value" } } ``` @@ -968,16 +968,16 @@ Follow these steps to view details of a vulnerability: | Field | Description | |:--------------------|:----------------------------------------------------------------------------------------| - | Description | Description of the vulnerability including what was modified. | + | Description | Description of the vulnerability including what was modified. | | Project | Namespace and project in which the vulnerability was detected. | | Method | HTTP method used to detect the vulnerability. | | URL | URL at which the vulnerability was detected. | - | Request | The HTTP request that caused the vulnerability. | + | Request | The HTTP request that caused the vulnerability. | | Unmodified Response | Response from an unmodified request. This is what a normal working response looks like. | - | Actual Response | Response received from test request. | - | Evidence | How we determined a vulnerability occurred. | - | Identifiers | The DAST API check used to find this vulnerability. | - | Severity | Severity of the vulnerability. | + | Actual Response | Response received from test request. | + | Evidence | How we determined a vulnerability occurred. | + | Identifiers | The DAST API check used to find this vulnerability. | + | Severity | Severity of the vulnerability. | | Scanner Type | Scanner used to perform testing. | ### Security Dashboard @@ -1105,8 +1105,46 @@ Profiles: - Name: XmlInjectionCheck ``` +## Running DAST API in an offline environment + +For self-managed GitLab instances in an environment with limited, restricted, or intermittent access to external resources through the internet, some adjustments are required for the DAST API testing job to successfully run. + +Steps: + +1. Host the Docker image in a local container registry. +1. Set the `SECURE_ANALYZERS_PREFIX` to the local container registry. + +The Docker image for DAST API must be pulled (downloaded) from the public registry and then pushed (imported) into a local registry. The GitLab container registry can be used to locally host the Docker image. This process can be performed using a special template. See [loading Docker images onto your offline host](../offline_deployments/index.md#loading-docker-images-onto-your-offline-host) for instructions. + +Once the Docker image is hosted locally, the `SECURE_ANALYZERS_PREFIX` variable is set with the location of the local registry. The variable must be set such that concatenating `/api-fuzzing:1` results in a valid image location. + +NOTE: +DAST API and API Fuzzing both use the same underlying Docker image `api-fuzzing:1`. + +For example, the below line sets a registry for the image `registry.gitlab.com/gitlab-org/security-products/analyzers/api-fuzzing:1`: + +`SECURE_ANALYZERS_PREFIX: "registry.gitlab.com/gitlab-org/security-products/analyzers"` + +NOTE: +Setting `SECURE_ANALYZERS_PREFIX` changes the Docker image registry location for all GitLab Secure templates. + +For more information, see [Offline environments](../offline_deployments/index.md). + ## Troubleshooting +### Error waiting for API Security 'http://127.0.0.1:5000' to become available + +A bug exists in versions of the DAST API analyzer prior to v1.6.196 that can cause a background process to fail under certain conditions. The solution is to update to a newer version of the DAST API analyzer. + +The version information can be found in the job details for the `dast_api` job. + +If the issue is occuring with versions v1.6.196 or greater, please contact Support and provide the following information: + +1. Reference this troubleshooting section and ask for the issue to be escalated to the Dynamic Analysis Team. +1. The full console output of the job. +1. The `gl-api-security-scanner.log` file available as a job artifact. In the right-hand panel of the job details page, select the **Browse** button. +1. The `dast_api` job definition from your `.gitlab-ci.yml` file. + ### Failed to start scanner session (version header not found) The DAST API engine outputs an error message when it cannot establish a connection with the scanner application component. The error message is shown in the job output window of the `dast_api` job. A common cause of this issue is changing the `DAST_API_API` variable from its default. @@ -1114,7 +1152,7 @@ The DAST API engine outputs an error message when it cannot establish a connecti **Error message** - In [GitLab 13.11 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/323939), `Failed to start scanner session (version header not found).` -- In GitLab 13.10 and earlier, `API Security version header not found. Are you sure that you are connecting to the API Security server?`. +- In GitLab 13.10 and earlier, `API Security version header not found. Are you sure that you are connecting to the API Security server?`. **Solution** diff --git a/doc/user/application_security/dependency_list/index.md b/doc/user/application_security/dependency_list/index.md index edfd0333d54..b0d8af2606f 100644 --- a/doc/user/application_security/dependency_list/index.md +++ b/doc/user/application_security/dependency_list/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Dependency list **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10075) in GitLab Ultimate 12.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10075) in GitLab 12.0. Use the dependency list to review your project's dependencies and key details about those dependencies, including their known vulnerabilities. It is a collection of dependencies in your project, including existing and new findings. @@ -66,7 +66,7 @@ Dependency paths are supported for the following package managers: ## Licenses -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10536) in GitLab Ultimate 12.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/10536) in GitLab 12.3. If the [License Compliance](../../compliance/license_compliance/index.md) CI job is configured, [discovered licenses](../../compliance/license_compliance/index.md#supported-languages-and-package-managers) are displayed on this page. diff --git a/doc/user/application_security/dependency_scanning/index.md b/doc/user/application_security/dependency_scanning/index.md index 1f840c96663..3c6db8c3ee9 100644 --- a/doc/user/application_security/dependency_scanning/index.md +++ b/doc/user/application_security/dependency_scanning/index.md @@ -89,12 +89,13 @@ table.supported-languages ul { } </style> +<!-- markdownlint-disable MD044 --> <table class="supported-languages"> <thead> <tr> <th>Language</th> + <th>Language Versions</th> <th>Package Manager</th> - <th>Package Manager Versions</th> <th>Supported files</th> <th>Analyzer</th> <th><a href="#how-multiple-files-are-processed">Processes multiple files?</a></th> @@ -103,8 +104,8 @@ table.supported-languages ul { <tbody> <tr> <td rowspan="2">Ruby</td> + <td rowspan="2">N/A</td> <td rowspan="2"><a href="https://bundler.io/">Bundler</a></td> - <td rowspan="2">Any</td> <td> <ul> <li><code>Gemfile.lock</code></li> @@ -121,16 +122,16 @@ table.supported-languages ul { </tr> <tr> <td>PHP</td> + <td>N/A</td> <td><a href="https://getcomposer.org/">Composer</a></td> - <td>Any</td> <td><code>composer.lock</code></td> <td><a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a></td> <td>Y</td> </tr> <tr> <td>C</td> + <td rowspan="2">N/A</td> <td rowspan="2"><a href="https://conan.io/">Conan</a></td> - <td rowspan="2">Any</td> <td rowspan="2"><a href="https://docs.conan.io/en/latest/versioning/lockfiles.html"><code>conan.lock</code></a></td> <td rowspan="2"><a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a></td> <td rowspan="2">Y</td> @@ -140,16 +141,16 @@ table.supported-languages ul { </tr> <tr> <td>Go</td> - <td><a href="https://golang.org/">Golang</a></td> - <td>Any</td> + <td>N/A</td> + <td><a href="https://golang.org/">Go</a></td> <td><code>go.sum</code></td> <td><a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a></td> <td>Y</td> </tr> <tr> <td rowspan="2">Java</td> - <td><a href="https://gradle.org/">Gradle</a><sup><b><a href="#notes-regarding-supported-languages-and-package-managers">1</a></b></sup></td> - <td>Any</td> + <td rowspan="2">8, 11, 13, 14, 15, or 16</td> + <td><a href="https://gradle.org/">Gradle</a><sup><b><a href="#notes-regarding-supported-languages-and-package-managers-1">1</a></b></sup></td> <td> <ul> <li><code>build.gradle</code></li> @@ -161,15 +162,14 @@ table.supported-languages ul { </tr> <tr> <td><a href="https://maven.apache.org/">Maven</a></td> - <td>Any</td> <td><code>pom.xml</code></td> <td><a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a></td> <td>N</td> </tr> <tr> <td rowspan="3">JavaScript</td> + <td rowspan="2">N/A</td> <td rowspan="2"><a href="https://www.npmjs.com/">npm</a></td> - <td rowspan="2">Any</td> <td> <ul> <li><code>package-lock.json</code></li> @@ -185,16 +185,16 @@ table.supported-languages ul { <td>N</td> </tr> <tr> + <td>N/A</td> <td><a href="https://classic.yarnpkg.com/en/">yarn</a></td> - <td>1.x</td> <td><code>yarn.lock</code></td> <td><a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a></td> <td>Y</td> </tr> <tr> <td>.NET</td> + <td rowspan="2">N/A</td> <td rowspan="2"><a href="https://www.nuget.org/">NuGet</a></td> - <td rowspan="2">>= 4.9</td> <td rowspan="2"><a href="https://docs.microsoft.com/en-us/nuget/consume-packages/package-references-in-project-files#enabling-lock-file"><code>packages.lock.json</code></a></td> <td rowspan="2"><a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a></td> <td rowspan="2">Y</td> @@ -204,15 +204,14 @@ table.supported-languages ul { </tr> <tr> <td rowspan="3">Python</td> + <td rowspan="3">3.6</td> <td><a href="https://setuptools.readthedocs.io/en/latest/">setuptools</a></td> - <td>Any</td> <td><code>setup.py</code></td> <td><a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a></td> <td>N</td> </tr> <tr> <td><a href="https://pip.pypa.io/en/stable/">pip</a></td> - <td>Any</td> <td> <ul> <li><code>requirements.txt</code></li> @@ -225,11 +224,10 @@ table.supported-languages ul { </tr> <tr> <td><a href="https://pipenv.pypa.io/en/latest/">Pipenv</a></td> - <td>Any</td> <td> <ul> <li><a href="https://pipenv.pypa.io/en/latest/basics/#example-pipfile-pipfile-lock"><code>Pipfile</code></a></li> - <li><a href="https://pipenv.pypa.io/en/latest/basics/#example-pipfile-pipfile-lock"><code>Pipfile.lock</code></a><sup><b><a href="#notes-regarding-supported-languages-and-package-managers">2</a></b></sup></li> + <li><a href="https://pipenv.pypa.io/en/latest/basics/#example-pipfile-pipfile-lock"><code>Pipfile.lock</code></a><sup><b><a href="#notes-regarding-supported-languages-and-package-managers-2">2</a></b></sup></li> </ul> </td> <td><a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a></td> @@ -237,8 +235,8 @@ table.supported-languages ul { </tr> <tr> <td>Scala</td> - <td><a href="https://www.scala-sbt.org/">sbt</a><sup><b><a href="#notes-regarding-supported-languages-and-package-managers">3</a></b></sup></td> - <td>Any</td> + <td>N/A</td> + <td><a href="https://www.scala-sbt.org/">sbt</a><sup><b><a href="#notes-regarding-supported-languages-and-package-managers-3">3</a></b></sup></td> <td><code>build.sbt</code></td> <td><a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a></td> <td>N</td> @@ -246,18 +244,91 @@ table.supported-languages ul { </tbody> </table> -### Notes regarding supported languages and package managers - -1. Although Gradle with Java 8 is supported, there are other issues such that Android project builds are not supported at this time. Please see the backlog issue [Android support for Dependency Scanning (gemnasium-maven)](https://gitlab.com/gitlab-org/gitlab/-/issues/336866) for more details. - -1. The presence of a `Pipfile.lock` file alone will _not_ trigger the analyzer; the presence of a `Pipfile` is still required in order -for the analyzer to be executed. However, if a `Pipfile.lock` file is found, it will be used by `Gemnasium` to scan the exact package -versions listed in this file. - - Support for `Pipfile.lock` files without requiring the presence of a `Pipfile` will be implemented in the following upcoming issue: - [Dependency Scanning of Pipfile.lock without installing project dependencies](https://gitlab.com/gitlab-org/gitlab/-/issues/299294). - -1. Support for [sbt](https://www.scala-sbt.org/) 1.3 and above was added in GitLab 13.9. +<ol> + <li> + <a id="notes-regarding-supported-languages-and-package-managers-1"></a> + <p> + Although Gradle with Java 8 is supported, there are other issues such that Android project builds are not supported at this time. + Please see the backlog issue <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/336866">Android support for Dependency + Scanning (gemnasium-maven)</a> for more details. + </p> + </li> + <li> + <a id="notes-regarding-supported-languages-and-package-managers-2"></a> + <p> + The presence of a <code>Pipfile.lock</code> file alone will <i>not</i> trigger the analyzer; the presence of a <code>Pipfile</code> is + still required in order for the analyzer to be executed. However, if a <code>Pipfile.lock</code> file is found, it will be used by + <code>Gemnasium</code> to scan the exact package versions listed in this file. + </p> + <p> + Support for <code>Pipfile.lock</code> files without requiring the presence of a <code>Pipfile</code> is tracked in + issue: <a href="https://gitlab.com/gitlab-org/gitlab/-/issues/299294">Dependency Scanning of Pipfile.lock without + installing project dependencies</a>. + </p> + </li> + <li> + <a id="notes-regarding-supported-languages-and-package-managers-3"></a> + <p> + Support for <a href="https://www.scala-sbt.org/">sbt</a> 1.3 and above was added in GitLab 13.9. + </p> + </li> +</ol> +<!-- markdownlint-enable MD044 --> + +### How analyzers obtain dependency information + +GitLab analyzers obtain dependency information using one of the following two methods: + +1. [Parsing lockfiles directly.](#obtaining-dependendency-information-by-parsing-lockfiles) +1. [Running a package manager or build tool to generate a dependency information file which is then parsed.](#obtaining-dependendency-information-by-running-a-package-manager-to-generate-a-parsable-file) + +#### Obtaining dependendency information by parsing lockfiles + +The following package managers use lockfiles that GitLab analyzers are capable of parsing directly: + +| Package Manager | Supported File Format Versions | Tested Versions | +| ------ | ------ | ------ | +| Bundler | N/A | [1.17.3](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/master/Gemfile.lock#L118), [2.1.4](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/bundler2-FREEZE/Gemfile.lock#L118) | +| Composer | N/A | [1.x](https://gitlab.com/gitlab-org/security-products/tests/php-composer/-/blob/master/composer.lock) | +| Conan | 0.4 | [1.x](https://gitlab.com/gitlab-org/security-products/tests/c-conan/-/blob/master/conan.lock) | +| Go | N/A | [1.x](https://gitlab.com/gitlab-org/security-products/tests/go-modules/-/blob/master/go.mod) | +| NuGet | v1 | [4.9](https://gitlab.com/gitlab-org/security-products/tests/csharp-nuget-dotnetcore/-/blob/master/src/web.api/packages.lock.json#L2) | +| npm | v1, v2 | [6.x](https://gitlab.com/gitlab-org/security-products/tests/js-npm/-/blob/master/package-lock.json#L4), [7.x](https://gitlab.com/gitlab-org/security-products/tests/js-npm/-/blob/lockfile-v2-FREEZE/package-lock.json#L4) | +| yarn | v1 | [1.x](https://gitlab.com/gitlab-org/security-products/tests/js-yarn/-/blob/master/yarn.lock) | + +#### Obtaining dependendency information by running a package manager to generate a parsable file + +To support the following package managers, the GitLab analyzers proceed in two steps: + +1. Execute the package manager or a specific task, to export the dependency information. +1. Parse the exported dependency information. + +| Package Manager | Preinstalled Versions | Tested Versions | +| ------ | ------ | ------ | +| Bundler | [2.1.4](https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit/-/blob/v2.11.3/Dockerfile#L15)<sup><b><a href="#exported-dependency-information-notes-1">1</a></b></sup> | [1.17.3](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/master/Gemfile.lock#L118), [2.1.4](https://gitlab.com/gitlab-org/security-products/tests/ruby-bundler/-/blob/bundler2-FREEZE/Gemfile.lock#L118) | +| sbt | [1.3.8](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L4) | [1.0.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L263), [1.1.4](https://gitlab.com/gitlab-org/security-products/tests/scala-sbt-multiproject/-/blob/main/project/build.properties#L1), [1.1.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L272), [1.2.8](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L281), [1.3.12](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L290), [1.4.6](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/master/.gitlab-ci.yml#L299) | +| Maven | [3.6.3](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L3) | [3.6.3](https://gitlab.com/gitlab-org/security-products/tests/java-maven/-/blob/master/pom.xml#L3) | +| Gradle | [6.7.1](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-maven/-/blob/v2.23.0/config/.tool-versions#L5) | [5.6.4](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/master/gradle/wrapper/gradle-wrapper.properties#L3), [6.5](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14/gradle/wrapper/gradle-wrapper.properties#L3), [6.7-rc-1](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-15/gradle/wrapper/gradle-wrapper.properties#L3), [6.9](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-14-gradle-6-9/gradle/wrapper/gradle-wrapper.properties#L3), [7.0-rc-2](https://gitlab.com/gitlab-org/security-products/tests/java-gradle/-/blob/java-16/gradle/wrapper/gradle-wrapper.properties#L3) | +| setuptools | [50.3.2](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L27) | | +| pip | [20.2.4](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/v2.29.9/Dockerfile#L26) | [20.x](https://gitlab.com/gitlab-org/security-products/tests/python-pip/-/blob/master/requirements.txt) | +| Pipenv | [2018.11.26](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/v2.18.4/requirements.txt#L13) | [2018.11.26](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/pipfile-lock-FREEZE/Pipfile.lock#L6)<sup><b><a href="#exported-dependency-information-notes-2">2</a></b></sup>, [2018.11.26](https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/blob/master/Pipfile) | + +<!-- markdownlint-disable MD044 --> +<ol> + <li> + <a id="exported-dependency-information-notes-1"></a> + <p> + The installed version of <code>Bundler</code> is only used for the <a href="https://gitlab.com/gitlab-org/security-products/analyzers/bundler-audit">bundler-audit</a> analyzer, and is not used for <a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">gemnasium</a> + </p> + </li> + <li> + <a id="exported-dependency-information-notes-2"></a> + <p> + This test confirms that if a <code>Pipfile.lock</code> file is found, it will be used by <a href="https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium">Gemnasium</a> to scan the exact package versions listed in this file. + </p> + </li> +</ol> +<!-- markdownlint-enable MD044 --> ### How analyzers are triggered @@ -321,15 +392,16 @@ We execute both analyzers because they use different sources of vulnerability da The analyzer for these languages supports multiple lockfiles. -### Future support for additional languages +### Support for additional languages -Plans are underway for supporting the following languages, dependency managers, and dependency files. For details, see the issue link for each. -For workarounds, see the [Troubleshooting section](#troubleshooting) +Support for additional languages, dependency managers, and dependency files are tracked in the following issues: | Package Managers | Languages | Supported files | Scan tools | Issue | | ------------------- | --------- | --------------- | ---------- | ----- | | [Poetry](https://python-poetry.org/) | Python | `poetry.lock` | [Gemnasium](https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium) | [GitLab#7006](https://gitlab.com/gitlab-org/gitlab/-/issues/7006) | +For workarounds, see the [Troubleshooting section](#troubleshooting). + ## Contribute your scanner The [Security Scanner Integration](../../../development/integrations/secure.md) documentation explains how to integrate other security scanners into GitLab. @@ -359,8 +431,8 @@ always take the latest dependency scanning artifact available. ### Enable Dependency Scanning via an automatic merge request -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4908) in GitLab 14.1. -> - [Enabled with `sec_dependency_scanning_ui_enable` flag](https://gitlab.com/gitlab-org/gitlab/-/issues/282533) for self-managed GitLab in GitLab 14.1 and is ready for production use. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4908) in GitLab 14.1 [with a flag](../../../administration/feature_flags.md) named `sec_dependency_scanning_ui_enable`. Enabled by default. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/282533) in GitLab 14.1. > - [Feature flag sec_dependency_scanning_ui_enable removed](https://gitlab.com/gitlab-org/gitlab/-/issues/326005) in GitLab 14.2. To enable Dependency Scanning in a project, you can create a merge request diff --git a/doc/user/application_security/iac_scanning/index.md b/doc/user/application_security/iac_scanning/index.md new file mode 100644 index 00000000000..a58a00a869b --- /dev/null +++ b/doc/user/application_security/iac_scanning/index.md @@ -0,0 +1,98 @@ +--- +stage: Secure +group: Static Analysis +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Infrastructure as Code (IaC) Scanning + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/6655) in GitLab 14.5. + +Infrastructure as Code (IaC) Scanning scans your IaC configuration files for known vulnerabilities. + +Currently, IaC scanning supports configuration files for Terraform, Ansible, AWS CloudFormation, and Kubernetes. + +## Requirements + +To run IaC scanning jobs, by default, you need GitLab Runner with the +[`docker`](https://docs.gitlab.com/runner/executors/docker.html) or +[`kubernetes`](https://docs.gitlab.com/runner/install/kubernetes.html) executor. +If you're using the shared runners on GitLab.com, this is enabled by default. + +WARNING: +Our IaC scanning jobs require a Linux container type. Windows containers are not yet supported. + +WARNING: +If you use your own runners, make sure the Docker version installed +is **not** `19.03.0`. See [troubleshooting information](../sast/index.md#error-response-from-daemon-error-processing-tar-file-docker-tar-relocation-error) for details. + +## Supported languages and frameworks + +GitLab IaC scanning supports a variety of IaC configuration files. Our IaC security scanners also feature automatic language detection which works even for mixed-language projects. If any supported configuration files are detected in project source code we automatically run the appropriate IaC analyzers. + +| Configuration File Type | Scan tool | Introduced in GitLab Version | +|------------------------------------------|----------------------------------|-------------------------------| +| Ansible | [kics](https://kics.io/) | 14.5 | +| AWS CloudFormation | [kics](https://kics.io/) | 14.5 | +| Kubernetes | [kics](https://kics.io/) | 14.5 | +| Terraform | [kics](https://kics.io/) | 14.5 | + +### Making IaC analyzers available to all GitLab tiers + +All open source (OSS) analyzers are availibile with the GitLab Free tier. Future propietary analyzers may be restricted to higher tiers. + +#### Summary of features per tier + +Different features are available in different [GitLab tiers](https://about.gitlab.com/pricing/), +as shown in the following table: + +| Capability | In Free | In Ultimate | +|:---------------------------------------------------------------------------------------|:--------------------|:-------------------| +| [Configure IaC Scanners](#configuration) v | **{check-circle}** | **{check-circle}** | +| View [JSON Report](#reports-json-format) | **{check-circle}** | **{check-circle}** | +| Presentation of JSON Report in Merge Request | **{dotted-circle}** | **{check-circle}** | +| [Address vulnerabilities](../../application_security/vulnerabilities/index.md) | **{dotted-circle}** | **{check-circle}** | +| [Access to Security Dashboard](../../application_security/security_dashboard/index.md) | **{dotted-circle}** | **{check-circle}** | + +## Contribute your scanner + +The [Security Scanner Integration](../../../development/integrations/secure.md) documentation explains how to integrate other security scanners into GitLab. + +## Configuration + +To configure IaC Scanning for a project you can: + +- [Configure IaC Scanning manually](#configure-iac-scanning-manually) +- [Enable IaC Scanning via an automatic merge request](#enable-iac-scanning-via-an-automatic-merge-request) + +### Configure IaC Scanning manually + +To enable IaC Scanning you must [include](../../../ci/yaml/index.md#includetemplate) the +[`SAST-IaC.latest.gitlab-ci.yml template`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST-IaC.latest.gitlab-ci.yml) provided as part of your GitLab installation. + +The included template creates IaC scanning jobs in your CI/CD pipeline and scans +your project's configuration files for possible vulnerabilities. + +The results are saved as a +[SAST report artifact](../../../ci/yaml/index.md#artifactsreportssast) +that you can download and analyze. + +### Enable IaC Scanning via an automatic merge request + +To enable IaC Scanning in a project, you can create a merge request +from the Security Configuration page: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Security & Compliance > Configuration**. +1. In the **Infrastructure as Code (IaC) Scanning** row, select **Configure via Merge Request**. + +This automatically creates a merge request with the changes necessary to enable IaC Scanning +that you can review and merge to complete the configuration. + +## Reports JSON format + +The IaC tool emits a JSON report file in the existing SAST report format. For more information, see the +[schema for this report](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/blob/master/dist/sast-report-format.json). + +The JSON report file can be downloaded from the CI pipelines page, or the +pipelines tab on merge requests by [setting `artifacts: paths`](../../../ci/yaml/index.md#artifactspaths) to `gl-sast-report.json`. For more information see [Downloading artifacts](../../../ci/pipelines/job_artifacts.md). diff --git a/doc/user/application_security/img/vulnerability-check_v14_2.png b/doc/user/application_security/img/vulnerability-check_v14_2.png Binary files differdeleted file mode 100644 index 655e43221c7..00000000000 --- a/doc/user/application_security/img/vulnerability-check_v14_2.png +++ /dev/null diff --git a/doc/user/application_security/index.md b/doc/user/application_security/index.md index 7b95769a81f..d5e801ced9c 100644 --- a/doc/user/application_security/index.md +++ b/doc/user/application_security/index.md @@ -31,19 +31,20 @@ For an overview of GitLab application security, see [Shifting Security Left](htt GitLab uses the following tools to scan and report known vulnerabilities found in your project. -| Secure scanning tool | Description | -|:-----------------------------------------------------------------------------|:-----------------------------------------------------------------------| -| [Container Scanning](container_scanning/index.md) **(ULTIMATE)** | Scan Docker containers for known vulnerabilities. | -| [Dependency List](dependency_list/index.md) **(ULTIMATE)** | View your project's dependencies and their known vulnerabilities. | -| [Dependency Scanning](dependency_scanning/index.md) **(ULTIMATE)** | Analyze your dependencies for known vulnerabilities. | -| [Dynamic Application Security Testing (DAST)](dast/index.md) **(ULTIMATE)** | Analyze running web applications for known vulnerabilities. | -| [DAST API](dast_api/index.md) **(ULTIMATE)** | Analyze running web APIs for known vulnerabilities. | -| [API fuzzing](api_fuzzing/index.md) **(ULTIMATE)** | Find unknown bugs and vulnerabilities in web APIs with fuzzing. | -| [Secret Detection](secret_detection/index.md) | Analyze Git history for leaked secrets. | -| [Security Dashboard](security_dashboard/index.md) **(ULTIMATE)** | View vulnerabilities in all your projects and groups. | -| [Static Application Security Testing (SAST)](sast/index.md) | Analyze source code for known vulnerabilities. | -| [Coverage fuzzing](coverage_fuzzing/index.md) **(ULTIMATE)** | Find unknown bugs and vulnerabilities with coverage-guided fuzzing. | -| [Cluster Image Scanning](cluster_image_scanning/index.md) **(ULTIMATE)** | Scan Kubernetes clusters for known vulnerabilities. | +| Secure scanning tool | Description | +| :------------------------------------------------------------- | :------------------------------------------------------------------ | +| [Container Scanning](container_scanning/index.md) | Scan Docker containers for known vulnerabilities. | +| [Dependency List](dependency_list/index.md) | View your project's dependencies and their known vulnerabilities. | +| [Dependency Scanning](dependency_scanning/index.md) | Analyze your dependencies for known vulnerabilities. | +| [Dynamic Application Security Testing (DAST)](dast/index.md) | Analyze running web applications for known vulnerabilities. | +| [DAST API](dast_api/index.md) | Analyze running web APIs for known vulnerabilities. | +| [API fuzzing](api_fuzzing/index.md) | Find unknown bugs and vulnerabilities in web APIs with fuzzing. | +| [Secret Detection](secret_detection/index.md) | Analyze Git history for leaked secrets. | +| [Security Dashboard](security_dashboard/index.md) | View vulnerabilities in all your projects and groups. | +| [Static Application Security Testing (SAST)](sast/index.md) | Analyze source code for known vulnerabilities. | +| [Infrastructure as Code (IaC) Scanning](iac_scanning/index.md) | Analyze your IaC coniguration files for known vulnerabilities. | +| [Coverage fuzzing](coverage_fuzzing/index.md) | Find unknown bugs and vulnerabilities with coverage-guided fuzzing. | +| [Cluster Image Scanning](cluster_image_scanning/index.md) | Scan Kubernetes clusters for known vulnerabilities. | ## Security scanning with Auto DevOps @@ -185,61 +186,51 @@ By default, the vulnerability report does not show vulnerabilities of `dismissed ## Security approvals in merge requests -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9928) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9928) in GitLab 12.2. -You can implement merge request approvals to require approval by selected users or a group when a -merge request would introduce one of the following security issues: +You can enforce an additional approval for merge requests that would introduce one of the following +security issues: -- A security vulnerability -- A software license compliance violation +- A security vulnerability. For more details, read + [Vulnerability-Check rule](#vulnerability-check-rule). +- A software license compliance violation. For more details, read + [Enabling license approvals within a project](../compliance/license_compliance/index.md#enabling-license-approvals-within-a-project). -When the Vulnerability-Check merge request rule is enabled, additional merge request approval +### Vulnerability-Check rule + +To prevent a merge request introducing a security vulnerability in a project, enable the +Vulnerability-Check rule. While this rule is enabled, additional merge request approval by +[eligible approvers](../project/merge_requests/approvals/rules.md#eligible-approvers) is required when the latest security report in a merge request: -- Contains vulnerabilities that are not present in the - target branch. Note that approval is still required for dismissed vulnerabilities. +- Contains vulnerabilities with states (for example, `previously detected`, `dismissed`) matching the rule's vulnerability states. Only `newly detected` will be considered if the target branch differs from the project default branch. - Contains vulnerabilities with severity levels (for example, `high`, `critical`, or `unknown`) matching the rule's severity levels. - Contains a vulnerability count higher than the rule allows. -- Is not generated during pipeline execution. +- Is not yet generated (until pipeline completion). An approval is optional when the security report: -- Contains no new vulnerabilities when compared to the target branch. +- Contains only vulnerabilities with states (for example, `newly detected`, `resolved`) **NOT** matching the rule's vulnerability states. - Contains only vulnerabilities with severity levels (for example, `low`, `medium`) **NOT** matching the rule's severity levels. - Contains a vulnerability count equal to or less than what the rule allows. -When the License-Check merge request rule is enabled, additional approval is required if a merge -request contains a denied license. For more details, see [Enabling license approvals within a project](../compliance/license_compliance/index.md#enabling-license-approvals-within-a-project). - -### Enable the Vulnerability-Check rule - -Prerequisites: - -- Maintainer or Owner [role](../permissions.md#project-members-permissions). +Project members assigned [at least the Maintainer role](../permissions.md#project-members-permissions) can enable or edit +the Vulnerability-Check rule. -For this approval group, you must set the number of approvals required to greater than zero. +#### Enable the Vulnerability-Check rule -Follow these steps to enable `Vulnerability-Check`: +To enable or edit the Vulnerability-Check rule: 1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > General**. 1. Expand **Merge request approvals**. -1. Select **Enable** or **Edit**. -1. Set the **Security scanners** that the rule applies to. -1. Select the **Target branch**. -1. Set the **Vulnerabilities allowed** to the number of vulnerabilities allowed before the rule is - triggered. -1. Set the **Severity levels** to the severity levels that the rule applies to. -1. Set the **Approvals required** to the number of approvals that the rule requires. -1. Select the users or groups to provide approval. +1. Select **Activate** or **Edit** of the Vulnerability-Check. +1. Complete the fields. **Approvals required** must be at least 1. 1. Select **Add approval rule**. -Once this group is added to your project, the approval rule is enabled for all merge requests. -Any code changes cause the approvals required to reset. - - +The approval rule is enabled for all merge requests. Any code changes reset the approvals required. ## Using private Maven repositories @@ -270,28 +261,44 @@ under your project's settings: </settings> ``` -## DAST On-Demand Scans +## Using a custom scanning stage -If you don't want scans running in your normal DevOps process you can use on-demand scans instead. For more details, see [on-demand scans](dast/index.md#on-demand-scans). This feature is only available for DAST. If you run an on-demand scan against the default branch, it is reported as a "successful pipeline" and these results are included in the security dashboard and vulnerability report. +When security scanning is enabled by including CI/CD templates as described in the +[Security scanning without Auto DevOps](#security-scanning-without-auto-devops) section, the scanning jobs +use the predefined `test` stage by default. If you specify a custom stage in your `.gitlab-ci.yml` file without +including a `test` stage, an error occurs. -## Security report validation +For example, the following attempts to use a `unit-tests` stage: -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321918) in GitLab 13.11. -> - Schema validation message [added](https://gitlab.com/gitlab-org/gitlab/-/issues/321730) in GitLab 14.0. +```yaml +include: + - template: Security/Dependency-Scanning.gitlab-ci.yml + - template: Security/License-Scanning.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml + - template: Security/Secret-Detection.gitlab-ci.yml -You can optionally enable validation of the security report artifacts based on the -[report schemas](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/tree/master/dist). -If you enable validation, GitLab validates the report artifacts before ingesting the vulnerabilities. -This prevents ingestion of broken vulnerability data into the database. +stages: + - unit-tests -In GitLab 14.0 and later, the pipeline's **Security** tab lists any report artifacts -that failed validation. Security report validation must first be enabled. +custom job: + stage: unit-tests + script: + - echo "custom job" +``` -### Enable security report validation +The above `.gitlab-ci.yml` causes a linting error: -To enable report artifacts validation, set the `VALIDATE_SCHEMA` environment variable to `"true"` for the jobs in the `.gitlab-ci.yml` file. +```plaintext +Found errors in your .gitlab-ci.yml: +- dependency_scanning job: chosen stage does not exist; available stages are .pre +- unit-tests +- .post +``` + +This error appears because the `test` stage used by the security scanning jobs isn't declared in the `.gitlab-ci.yml` file. +To fix this issue, you can either: -For example, the configuration below enables validation for only the `sast` job: +- Add a `test` stage in your `.gitlab-ci.yml`: ```yaml include: @@ -301,26 +308,98 @@ For example, the configuration below enables validation for only the `sast` job: - template: Security/Secret-Detection.gitlab-ci.yml stages: - - security-scan + - test + - unit-tests + + custom job: + stage: unit-tests + script: + - echo "custom job" + ``` + +- Override the default stage of each security job. For example, to use a pre-defined stage named `unit-tests`: + + ```yaml + include: + - template: Security/Dependency-Scanning.gitlab-ci.yml + - template: Security/License-Scanning.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml + - template: Security/Secret-Detection.gitlab-ci.yml + + stages: + - unit-tests dependency_scanning: - stage: security-scan + stage: unit-tests license_scanning: - stage: security-scan + stage: unit-tests sast: - stage: security-scan - variables: - VALIDATE_SCHEMA: "true" + stage: unit-tests .secret-analyzer: - stage: security-scan + stage: unit-tests + + custom job: + stage: unit-tests + script: + - echo "custom job" ``` -## Interacting with findings and vulnerabilities +Learn more on overriding security jobs: + +- [Overriding SAST jobs](sast/index.md#overriding-sast-jobs). +- [Overriding Dependency Scanning jobs](dependency_scanning/index.md#overriding-dependency-scanning-jobs). +- [Overriding Container Scanning jobs](container_scanning/index.md#overriding-the-container-scanning-template). +- [Overriding Secret Detection jobs](secret_detection/index.md#customizing-settings). +- [Overriding DAST jobs](dast/index.md#customize-dast-settings). +- [Overriding License Compliance jobs](../compliance/license_compliance/index.md#overriding-the-template). + +All the security scanning tools define their stage, so this error can occur with all of them. + +## Security report validation + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/321918) in GitLab 13.11. +> - Schema validation message [added](https://gitlab.com/gitlab-org/gitlab/-/issues/321730) in GitLab 14.0. + +You can enforce validation of the security report artifacts before ingesting the vulnerabilities. +This prevents ingestion of broken vulnerability data into the database. GitLab validates the +artifacts based on the [report schemas](https://gitlab.com/gitlab-org/security-products/security-report-schemas/-/tree/master/dist). + +In GitLab 14.0 and later, when artifact validation is enabled, the pipeline's **Security** tab lists +any report artifacts that failed validation. + +### Enable security report validation + +To enable report artifacts validation, set the `VALIDATE_SCHEMA` environment variable to `"true"` +for the desired jobs in the `.gitlab-ci.yml` file. -There are a variety of locations and ways to interact with the results of the security scanning tools: +For example, to enable validation for only the `sast` job: + +```yaml +include: + - template: Security/Dependency-Scanning.gitlab-ci.yml + - template: Security/License-Scanning.gitlab-ci.yml + - template: Security/SAST.gitlab-ci.yml + - template: Security/Secret-Detection.gitlab-ci.yml +stages: + - security-scan +dependency_scanning: + stage: security-scan +license_scanning: + stage: security-scan +sast: + stage: security-scan + variables: + VALIDATE_SCHEMA: "true" +.secret-analyzer: + stage: security-scan +``` + +## Interact with findings and vulnerabilities + +You can interact with the results of the security scanning tools in several locations: - [Scan information in merge requests](#view-security-scan-information-in-merge-requests) - [Project Security Dashboard](security_dashboard/#project-security-dashboard) @@ -331,13 +410,33 @@ There are a variety of locations and ways to interact with the results of the se - [Vulnerability Pages](vulnerabilities/index.md) - [Dependency List](dependency_list/index.md) -For more details about which findings or vulnerabilities you can view in each of those locations, select the respective link. Each page details the ways in which you can interact with the findings and vulnerabilities. As an example, in most cases findings start out as _detected_ status. You have the option to: +For more details about which findings or vulnerabilities you can view in each of those locations, +select the respective link. Each page details the ways in which you can interact with the findings +and vulnerabilities. As an example, in most cases findings start out as _detected_ status. + +You have the option to: - Change the status. - Create an issue. - Link it to an existing issue. - [Resolve the vulnerability](vulnerabilities/index.md#resolve-a-vulnerability), if a solution is known. +## Security scanning configuration tips + +Each GitLab security scanning tool has a default +[CI/CD configuration file](https://gitlab.com/gitlab-org/gitlab/-/tree/master/lib/gitlab/ci/templates/Security), +also known as a _template_. + +When customizing the configuration: + +- [Include](../../ci/yaml/index.md#include) the scanning tool's CI/CD template. Don't _copy_ the content + of the template. +- Use the [stable](../../development/cicd/templates.md#stable-version) version of each template + for production workflows. The stable version changes less often, and breaking changes are only + made between major GitLab versions. The [latest](../../development/cicd/templates.md#latest-version) + version contains the most recent changes, but may have significant changes between minor GitLab versions. +- Only override values in the template as needed. All other values are inherited from the template. + ## Troubleshooting ### Secure job failing with exit code 1 @@ -352,8 +451,8 @@ variables: ### Outdated security reports -When a security report generated for a merge request becomes outdated, the merge request shows a warning -message in the security widget and prompts you to take an appropriate action. +When a security report generated for a merge request becomes outdated, the merge request shows a +warning message in the security widget and prompts you to take an appropriate action. This can happen in two scenarios: @@ -362,73 +461,28 @@ This can happen in two scenarios: #### Source branch is behind the target branch -This means the most recent common ancestor commit between the target branch and the source branch is -not the most recent commit on the target branch. This is by far the most common situation. +A security report can be out of date when the most recent common ancestor commit between the +target branch and the source branch is not the most recent commit on the target branch. -In this case you must rebase or merge to incorporate the changes from the target branch. +To fix this issue, rebase or merge to incorporate the changes from the target branch.  #### Target branch security report is out of date -This can happen for many reasons, including failed jobs or new advisories. When the merge request shows that a -security report is out of date, you must run a new pipeline on the target branch. -You can do it quickly by following the hyperlink given to run a new pipeline. +This can happen for many reasons, including failed jobs or new advisories. When the merge request +shows that a security report is out of date, you must run a new pipeline on the target branch. +Select **new pipeline** to run a new pipeline.  -### Getting error message `sast job: stage parameter should be [some stage name here]` - -When [including](../../ci/yaml/index.md#includetemplate) a `.gitlab-ci.yml` template -like [`SAST.gitlab-ci.yml`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Security/SAST.gitlab-ci.yml), -the following error may occur, depending on your GitLab CI/CD configuration: - -```plaintext -Found errors in your .gitlab-ci.yml: - -* sast job: stage parameter should be unit-tests -``` - -This error appears when the included job's stage (named `test`) isn't declared in `.gitlab-ci.yml`. -To fix this issue, you can either: - -- Add a `test` stage in your `.gitlab-ci.yml`. -- Override the default stage of each security job. For example, to use a pre-defined stage name `unit-tests`: - - ```yaml - include: - - template: Security/Dependency-Scanning.gitlab-ci.yml - - template: Security/License-Scanning.gitlab-ci.yml - - template: Security/SAST.gitlab-ci.yml - - template: Security/Secret-Detection.gitlab-ci.yml - - stages: - - unit-tests - - dependency_scanning: - stage: unit-tests - - license_scanning: - stage: unit-tests - - sast: - stage: unit-tests - - .secret-analyzer: - stage: unit-tests - ``` - -[Learn more on overriding SAST jobs](sast/index.md#overriding-sast-jobs). -All the security scanning tools define their stage, so this error can occur with all of them. - ### Getting warning messages `… report.json: no matching files` -This is often followed by the [error `No files to upload`](../../ci/pipelines/job_artifacts.md#error-message-no-files-to-upload), -and preceded by other errors or warnings that indicate why the JSON report wasn't generated. Please -check the entire job log for such messages. If you don't find these messages, retry the failed job -after setting `SECURE_LOG_LEVEL: "debug"` as a -[custom CI/CD variable](../../ci/variables/index.md#custom-cicd-variables). -This provides useful information to investigate further. +This message is often followed by the [error `No files to upload`](../../ci/pipelines/job_artifacts.md#error-message-no-files-to-upload), +and preceded by other errors or warnings that indicate why the JSON report wasn't generated. Check +the entire job log for such messages. If you don't find these messages, retry the failed job after +setting `SECURE_LOG_LEVEL: "debug"` as a [custom CI/CD variable](../../ci/variables/index.md#custom-cicd-variables). +This provides extra information to investigate further. ### Getting error message `sast job: config key may not be used with 'rules': only/except` @@ -526,23 +580,24 @@ involve pinning to the previous template versions, for example: ``` Additionally, we provide a dedicated project containing the versioned legacy templates. -This can be useful for offline setups or anyone wishing to use [Auto DevOps](../../topics/autodevops/index.md). +This can be used for offline setups or anyone wishing to use [Auto DevOps](../../topics/autodevops/index.md). Instructions are available in the [legacy template project](https://gitlab.com/gitlab-org/auto-devops-v12-10). #### Vulnerabilities are found, but the job succeeds. How can I have a pipeline fail instead? -This is the current default behavior, because the job's status indicates success or failure of the analyzer itself. -Analyzer results are displayed in the [job logs](../../ci/jobs/index.md#expand-and-collapse-job-log-sections), -[Merge Request widget](#view-security-scan-information-in-merge-requests) -or [Security Dashboard](security_dashboard/index.md). +In these circumstances, that the job succeeds is the default behavior. The job's status indicates +success or failure of the analyzer itself. Analyzer results are displayed in the +[job logs](../../ci/jobs/index.md#expand-and-collapse-job-log-sections), +[Merge Request widget](#view-security-scan-information-in-merge-requests) or +[Security Dashboard](security_dashboard/index.md). ### Error: job `is used for configuration only, and its script should not be executed` [Changes made in GitLab 13.4](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41260) to the `Security/Dependency-Scanning.gitlab-ci.yml` and `Security/SAST.gitlab-ci.yml` templates mean that if you enable the `sast` or `dependency_scanning` jobs by setting the `rules` attribute, -they will fail with the error `(job) is used for configuration only, and its script should not be executed`. +they fail with the error `(job) is used for configuration only, and its script should not be executed`. The `sast` or `dependency_scanning` stanzas can be used to make changes to all SAST or Dependency Scanning, such as changing `variables` or the `stage`, but they cannot be used to define shared `rules`. diff --git a/doc/user/application_security/offline_deployments/index.md b/doc/user/application_security/offline_deployments/index.md index cdf54070d69..915e43d0fa5 100644 --- a/doc/user/application_security/offline_deployments/index.md +++ b/doc/user/application_security/offline_deployments/index.md @@ -87,7 +87,9 @@ above. You can find more information at each of the pages below: - [Container scanning offline directions](../container_scanning/index.md#running-container-scanning-in-an-offline-environment) - [SAST offline directions](../sast/index.md#running-sast-in-an-offline-environment) +- [Secret Detection offline directions](../secret_detection/#running-secret-detection-in-an-offline-environment) - [DAST offline directions](../dast/run_dast_offline.md#run-dast-in-an-offline-environment) +- [API Fuzzing offline directions](../api_fuzzing/#running-api-fuzzing-in-an-offline-environment) - [License Compliance offline directions](../../compliance/license_compliance/index.md#running-license-compliance-in-an-offline-environment) - [Dependency Scanning offline directions](../dependency_scanning/index.md#running-dependency-scanning-in-an-offline-environment) diff --git a/doc/user/application_security/policies/index.md b/doc/user/application_security/policies/index.md index e1ddb3167ff..4d8be411dc5 100644 --- a/doc/user/application_security/policies/index.md +++ b/doc/user/application_security/policies/index.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Policies **(ULTIMATE)** -> - Introduced in GitLab Ultimate 13.10 [with a flag](https://gitlab.com/groups/gitlab-org/-/epics/5329) named `security_orchestration_policies_configuration`. Disabled by default. -> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/321258) in GitLab Ultimate 14.3. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5329) in GitLab 13.10 with a flag named `security_orchestration_policies_configuration`. Disabled by default. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/321258) in GitLab 14.3. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/321258) in GitLab 14.4. Policies in GitLab provide security teams a way to require scans of their choice to be run @@ -49,7 +49,7 @@ users must make changes by following the ## Policy editor -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3403) in GitLab Ultimate 13.4. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3403) in GitLab 13.4. You can use the policy editor to create, edit, and delete policies: @@ -79,7 +79,7 @@ mode to fix your policy before Rule mode is available again. ## Container Network Policy -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32365) in GitLab Ultimate 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32365) in GitLab 12.9. The **Container Network Policy** section provides packet flow metrics for your application's Kubernetes namespace. This section has the following @@ -118,9 +118,9 @@ examining the Cilium logs: kubectl -n gitlab-managed-apps logs -l k8s-app=cilium -c cilium-monitor ``` -### Change the enforcement status +### Change the status -To change a network policy's enforcement status: +To change a network policy's status: - Select the network policy you want to update. - Select **Edit policy**. @@ -154,12 +154,12 @@ at the bottom of the editor. ### Configure a Network Policy Alert -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3438) and [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/287676) in GitLab Ultimate 13.9. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3438) and [enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/287676) in GitLab 13.9. > - The feature flag was removed and the Threat Monitoring Alerts Project was [made generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/287676) in GitLab 14.0. You can use policy alerts to track your policy's impact. Alerts are only available if you've [installed](../../clusters/agent/repository.md) -and [configured](../../clusters/agent/index.md#create-an-agent-record-in-gitlab) +and [configured](../../clusters/agent/install/index.md#create-an-agent-record-in-gitlab) a Kubernetes Agent for this project. There are two ways to create policy alerts: diff --git a/doc/user/application_security/sast/analyzers.md b/doc/user/application_security/sast/analyzers.md index d399dcaf4a9..06c57e68121 100644 --- a/doc/user/application_security/sast/analyzers.md +++ b/doc/user/application_security/sast/analyzers.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # SAST Analyzers **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3775) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.3. -> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3775) in GitLab 10.3. +> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/2098) from GitLab Ultimate to GitLab Free in 13.3. SAST relies on underlying third party tools that are wrapped into what we call "Analyzers". An analyzer is a diff --git a/doc/user/application_security/sast/index.md b/doc/user/application_security/sast/index.md index 7ffefd34e40..af8585c6a18 100644 --- a/doc/user/application_security/sast/index.md +++ b/doc/user/application_security/sast/index.md @@ -7,8 +7,8 @@ type: reference, howto # Static Application Security Testing (SAST) **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3775) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.3. -> - All open source (OSS) analyzers were moved to GitLab Free in GitLab 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3775) in GitLab 10.3. +> - All open source (OSS) analyzers were moved from GitLab Ultimate to GitLab Free in GitLab 13.3. NOTE: The whitepaper ["A Seismic Shift in Application Security"](https://about.gitlab.com/resources/whitepaper-seismic-shift-application-security/) @@ -16,11 +16,10 @@ explains how 4 of the top 6 attacks were application based. Download it to learn organization. If you're using [GitLab CI/CD](../../../ci/index.md), you can use Static Application Security -Testing (SAST) to check your source code for known vulnerabilities. When a pipeline completes, -the results of the SAST analysis are processed and shown in the pipeline's Security tab. If the -pipeline is associated with a merge request, the SAST analysis is compared with the results of +Testing (SAST) to check your source code for known vulnerabilities. +If the pipeline is associated with a merge request, the SAST analysis is compared with the results of the target branch's analysis (if available). The results of that comparison are shown in the merge -request. **(ULTIMATE)** If the pipeline is running from the default branch, the results of the SAST +request. If the pipeline is running from the default branch, the results of the SAST analysis are available in the [security dashboards](../security_dashboard/index.md).  @@ -197,7 +196,7 @@ Use the method that best meets your needs. - [Configure SAST in the UI with default settings](#configure-sast-in-the-ui-with-default-settings) - [Configure SAST in the UI with customizations](#configure-sast-in-the-ui-with-customizations) -### Configure SAST in the UI with default settings **(FREE)** +### Configure SAST in the UI with default settings > [Introduced](https://about.gitlab.com/releases/2021/02/22/gitlab-13-9-released/#security-configuration-page-for-all-users) in GitLab 13.9 @@ -217,9 +216,9 @@ successfully, and an error may occur. ### Configure SAST in the UI with customizations **(ULTIMATE)** -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3659) in GitLab Ultimate 13.3. -> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/232862) in GitLab Ultimate 13.4. -> - [Improved](https://gitlab.com/groups/gitlab-org/-/epics/3635) in GitLab Ultimate 13.5. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3659) in GitLab 13.3. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/232862) in GitLab 13.4. +> - [Improved](https://gitlab.com/groups/gitlab-org/-/epics/3635) in GitLab 13.5. To enable and configure SAST with customizations: @@ -402,7 +401,7 @@ To create a custom ruleset: ### False Positive Detection **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292686) in GitLab 14.2. +> Introduced in GitLab 14.2. Vulnerabilities that have been detected and are false positives will be flagged as false positives in the security dashboard. @@ -423,7 +422,7 @@ Read more on [how to use private Maven repositories](../index.md#using-private-m ### Enabling Kubesec analyzer -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12752) in GitLab Ultimate 12.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/12752) in GitLab 12.6. You need to set `SCAN_KUBERNETES_MANIFESTS` to `"true"` to enable the Kubesec analyzer. In `.gitlab-ci.yml`, define: @@ -569,7 +568,7 @@ Some analyzers can be customized with CI/CD variables. #### Custom CI/CD variables -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18193) in GitLab Ultimate 12.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/18193) in GitLab 12.5. In addition to the aforementioned SAST configuration CI/CD variables, all [custom variables](../../../ci/variables/index.md#custom-cicd-variables) are propagated diff --git a/doc/user/application_security/secret_detection/index.md b/doc/user/application_security/secret_detection/index.md index 5933496ea00..140f660d729 100644 --- a/doc/user/application_security/secret_detection/index.md +++ b/doc/user/application_security/secret_detection/index.md @@ -7,8 +7,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Secret Detection **(FREE)** -> - [Introduced](https://about.gitlab.com/releases/2019/03/22/gitlab-11-9-released/#detect-secrets-and-credentials-in-the-repository) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.9. -> - Made [available in all tiers](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) in 13.3. +> - [Introduced](https://about.gitlab.com/releases/2019/03/22/gitlab-11-9-released/#detect-secrets-and-credentials-in-the-repository) in GitLab 11.9. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/222788) from GitLab Ultimate to GitLab Free in 13.3. A recurring problem when developing applications is that developers may unintentionally commit secrets and credentials to their remote repositories. If other people have access to the source, @@ -138,9 +138,9 @@ The results are saved as a that you can later download and analyze. Due to implementation limitations, we always take the latest Secret Detection artifact available. -### Enable Secret Detection via an automatic merge request **(FREE)** +### Enable Secret Detection via an automatic merge request -> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4496) in GitLab 13.11, behind a feature flag, enabled by default. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4496) in GitLab 13.11, deployed behind a feature flag, enabled by default. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/329886) in GitLab 14.1. To enable Secret Detection in a project, you can create a merge request @@ -165,7 +165,7 @@ by using the [`variables`](../../../ci/yaml/index.md#variables) parameter in `.gitlab-ci.yml`. To override a job definition, (for example, change properties like `variables` or `dependencies`), -declare a job with the same name as the SAST job to override. Place this new job after the template +declare a job with the same name as the secret detection job to override. Place this new job after the template inclusion and specify any additional keys under it. WARNING: @@ -202,8 +202,9 @@ Secret Detection can be customized by defining available CI/CD variables: | CI/CD variable | Default value | Description | |-----------------------------------|---------------|-------------| -| `SECRET_DETECTION_COMMIT_FROM` | - | The commit a Gitleaks scan starts at. | -| `SECRET_DETECTION_COMMIT_TO` | - | The commit a Gitleaks scan ends at. | +| `SECRET_DETECTION_COMMIT_FROM` | - | The commit a Gitleaks scan starts at. [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/243564) in GitLab 13.5. Replaced with `SECRET_DETECTION_COMMITS`. | +| `SECRET_DETECTION_COMMIT_TO` | - | The commit a Gitleaks scan ends at. [Removed](https://gitlab.com/gitlab-org/gitlab/-/issues/243564) in GitLab 13.5. Replaced with `SECRET_DETECTION_COMMITS`. | +| `SECRET_DETECTION_COMMITS` | - | The list of commits that Gitleaks should scan. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/243564) in GitLab 13.5. | | `SECRET_DETECTION_EXCLUDED_PATHS` | "" | Exclude vulnerabilities from output based on the paths. This is a comma-separated list of patterns. Patterns can be globs, or file or folder paths (for example, `doc,spec` ). Parent directories also match patterns. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/225273) in GitLab 13.3. | | `SECRET_DETECTION_HISTORIC_SCAN` | false | Flag to enable a historic Gitleaks scan. | @@ -348,6 +349,22 @@ For details on saving and transporting Docker images as a file, see Docker's doc [`docker save`](https://docs.docker.com/engine/reference/commandline/save/), [`docker load`](https://docs.docker.com/engine/reference/commandline/load/), [`docker export`](https://docs.docker.com/engine/reference/commandline/export/), and [`docker import`](https://docs.docker.com/engine/reference/commandline/import/). +### Set Secret Detection CI/CD variables to use the local Secret Detection analyzer container image + +Add the following configuration to your `.gitlab-ci.yml` file. You must replace +`SECURE_ANALYZERS_PREFIX` to refer to your local Docker container registry: + +```yaml +include: + - template: Security/Secret-Detection.gitlab-ci.yml + +variables: + SECURE_ANALYZERS_PREFIX: "localhost:5000/analyzers" +``` + +The Secret Detection job should now use the local copy of the Secret Detection analyzer Docker image to scan your code and generate +security reports without requiring internet access. + #### If support for Custom Certificate Authorities are needed Support for custom certificate authorities was introduced in the following versions. @@ -371,22 +388,6 @@ variables: The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a [custom variable in the UI](../../../ci/variables/index.md#custom-cicd-variables), either as a `file`, which requires the path to the certificate, or as a variable, which requires the text representation of the certificate. -### Set Secret Detection CI/CD variables to use local Secret Detection analyzer - -Add the following configuration to your `.gitlab-ci.yml` file. You must replace -`SECURE_ANALYZERS_PREFIX` to refer to your local Docker container registry: - -```yaml -include: - - template: Security/Secret-Detection.gitlab-ci.yml - -variables: - SECURE_ANALYZERS_PREFIX: "localhost:5000/analyzers" -``` - -The Secret Detection job should now use local copies of the Secret Detection analyzer to scan your code and generate -security reports without requiring internet access. - ## Troubleshooting ### Getting warning message `gl-secret-detection-report.json: no matching files` diff --git a/doc/user/application_security/security_dashboard/img/pipeline_security_dashboard_v14_2.png b/doc/user/application_security/security_dashboard/img/pipeline_security_dashboard_v14_2.png Binary files differdeleted file mode 100644 index 52249cef343..00000000000 --- a/doc/user/application_security/security_dashboard/img/pipeline_security_dashboard_v14_2.png +++ /dev/null diff --git a/doc/user/application_security/security_dashboard/img/pipeline_security_dashboard_v14_4.png b/doc/user/application_security/security_dashboard/img/pipeline_security_dashboard_v14_4.png Binary files differnew file mode 100644 index 00000000000..ac123d2b528 --- /dev/null +++ b/doc/user/application_security/security_dashboard/img/pipeline_security_dashboard_v14_4.png diff --git a/doc/user/application_security/security_dashboard/index.md b/doc/user/application_security/security_dashboard/index.md index c78179e9693..87875ec15ba 100644 --- a/doc/user/application_security/security_dashboard/index.md +++ b/doc/user/application_security/security_dashboard/index.md @@ -51,7 +51,7 @@ The security dashboard and vulnerability report displays information about vulne At the pipeline level, the Security section displays the vulnerabilities present in the branch of the project the pipeline ran against. - + Visit the page for any pipeline that ran any of the [supported reports](#supported-reports). To view the pipeline's security findings, select the **Security** tab when viewing the pipeline. @@ -64,11 +64,15 @@ the analyzer outputs an ### Scan details -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3728) in GitLab 13.10. +> - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3728) in GitLab 13.10. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/333660) in GitLab 14.2. -The **Scan details** section lists the scans run in the pipeline and the total number of -vulnerabilities per scan. For the DAST scan, select **Download scanned resources** to download a -CSV file containing details of the resources scanned. +The **Scan details** section lists the scans run in the pipeline and the total number of vulnerabilities +per scan. + +You can download the JSON artifacts from each security scan. Select **Download results** then +select the JSON artifact. Additionally for the DAST scan, from the **Download results** dropdown select +**Download scanned resources** to download a CSV file containing details of the resources scanned. ## Project Security Dashboard diff --git a/doc/user/application_security/threat_monitoring/index.md b/doc/user/application_security/threat_monitoring/index.md index 79f202a6edb..ae5f6ba0fe1 100644 --- a/doc/user/application_security/threat_monitoring/index.md +++ b/doc/user/application_security/threat_monitoring/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Threat Monitoring **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14707) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14707) in GitLab 12.9. The **Threat Monitoring** page provides alerts and metrics for the GitLab application runtime security features. You can access @@ -20,7 +20,7 @@ GitLab supports statistics for the following security features: ## Container Network Policy Alert list -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3438) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.9. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3438) in GitLab 13.9. The policy alert list displays your policy's alert activity. You can sort the list by these columns: @@ -44,5 +44,3 @@ Clicking an alert's row opens the alert drawer, which shows more information abo can also create an incident from the alert and update the alert status in the alert drawer. Clicking an alert's name takes the user to the [alert details page](../../../operations/incident_management/alerts.md#alert-details-page). - -For information on work in progress for the alerts dashboard, see [this epic](https://gitlab.com/groups/gitlab-org/-/epics/5041). diff --git a/doc/user/application_security/vulnerabilities/index.md b/doc/user/application_security/vulnerabilities/index.md index 9ebecc67704..7bdc8cc8479 100644 --- a/doc/user/application_security/vulnerabilities/index.md +++ b/doc/user/application_security/vulnerabilities/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Vulnerability Pages **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13561) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13561) in GitLab 13.0. Each vulnerability in a project has a Vulnerability Page. This page contains details of the vulnerability. The details included vary according to the type of vulnerability. Details of each @@ -20,6 +20,9 @@ vulnerability include: - Linked issues - Actions log +In GitLab 14.3 and later, if the scanner determined the vulnerability to be a false positive, an +alert message is included at the top of the vulnerability's page. + On the vulnerability's page, you can: - [Change the vulnerability's status](#change-vulnerability-status). diff --git a/doc/user/application_security/vulnerability_report/img/group_vulnerability_report_v14_2.png b/doc/user/application_security/vulnerability_report/img/group_vulnerability_report_v14_2.png Binary files differdeleted file mode 100644 index 44c689eda3e..00000000000 --- a/doc/user/application_security/vulnerability_report/img/group_vulnerability_report_v14_2.png +++ /dev/null diff --git a/doc/user/application_security/vulnerability_report/img/project_level_vulnerability_report_v14_5.png b/doc/user/application_security/vulnerability_report/img/project_level_vulnerability_report_v14_5.png Binary files differnew file mode 100644 index 00000000000..ac996fa32db --- /dev/null +++ b/doc/user/application_security/vulnerability_report/img/project_level_vulnerability_report_v14_5.png diff --git a/doc/user/application_security/vulnerability_report/index.md b/doc/user/application_security/vulnerability_report/index.md index f5b0194c320..d13647937a2 100644 --- a/doc/user/application_security/vulnerability_report/index.md +++ b/doc/user/application_security/vulnerability_report/index.md @@ -16,7 +16,16 @@ At all levels, the Vulnerability Report contains: - Filters for common vulnerability attributes. - Details of each vulnerability, presented in tabular layout. - +The **Activity** column contains icons to indicate the activity, if any, taken on the vulnerability +in that row: + +- Issues **{issues}**: Links to issues created for the vulnerability. For more details, read + [Create an issue for a vulnerability](../vulnerabilities/index.md#create-an-issue-for-a-vulnerability). +- Wrench **{admin}**: The vulnerability has been remediated. +- False positive **{false-positive}**: The scanner determined this vulnerability to be a false + positive. + + ## Project-level Vulnerability Report @@ -151,7 +160,7 @@ To change the status of vulnerabilities in the table: ### Change status of multiple vulnerabilities -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35816) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/35816) in GitLab 12.9. You can change the status of multiple vulnerabilities at once: @@ -162,8 +171,8 @@ You can change the status of multiple vulnerabilities at once: ## Export vulnerability details -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213014) in the Security Center (previously known as the Instance Security Dashboard) and project-level Vulnerability Report (previously known as the Project Security Dashboard) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.0. -> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/213013) to the group-level Vulnerability Report in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213014) in the Security Center (previously known as the Instance Security Dashboard) and project-level Vulnerability Report (previously known as the Project Security Dashboard) in GitLab 13.0. +> - [Added](https://gitlab.com/gitlab-org/gitlab/-/issues/213013) to the group-level Vulnerability Report in GitLab 13.1. You can export details of the vulnerabilities listed in the Vulnerability Report. The export format is CSV (comma separated values). Note that all vulnerabilities are included because filters don't @@ -197,7 +206,7 @@ thousands of vulnerabilities. Don't close the page until the download finishes. ## Dismiss a vulnerability -> The option of adding a dismissal reason was introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. +> The option of adding a dismissal reason was introduced in GitLab 12.0. You can dismiss a vulnerability for the entire project: diff --git a/doc/user/clusters/agent/ci_cd_tunnel.md b/doc/user/clusters/agent/ci_cd_tunnel.md index 6c8b7c95771..0dfdb37dc1f 100644 --- a/doc/user/clusters/agent/ci_cd_tunnel.md +++ b/doc/user/clusters/agent/ci_cd_tunnel.md @@ -4,15 +4,13 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# CI/CD Tunnel **(PREMIUM)** +# CI/CD Tunnel **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327409) in GitLab 14.1. > - The pre-configured `KUBECONFIG` was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/324275) in GitLab 14.2. > - The ability to authorize groups was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. - -WARNING: -The CI/CD Tunnel is not supported for GitLab self-managed instances installed via Omnibus. We -plan to [add support for Omnibus](https://gitlab.com/gitlab-org/gitlab/-/issues/324272) in the future. +> - [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) to GitLab Free in 14.5. +> - Support for Omnibus installations was [introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/5686) in GitLab 14.5. The CI/CD Tunnel enables users to access Kubernetes clusters from GitLab CI/CD jobs even if there is no network connectivity between GitLab Runner and a cluster. GitLab Runner does not have to be running in the same cluster. @@ -21,11 +19,11 @@ Only CI/CD jobs set in the configuration project can access one of the configure ## Prerequisites -- A running [`kas` instance](index.md#set-up-the-kubernetes-agent-server). -- A [configuration repository](index.md#define-a-configuration-repository) with an Agent config file +- A running [`kas` instance](install/index.md#set-up-the-kubernetes-agent-server). +- A [configuration repository](install/index.md#define-a-configuration-repository) with an Agent config file installed (`.gitlab/agents/<agent-name>/config.yaml`). -- An [Agent record](index.md#create-an-agent-record-in-gitlab). -- The Agent [installed in the cluster](index.md#install-the-agent-into-the-cluster). +- An [Agent record](install/index.md#create-an-agent-record-in-gitlab). +- The Agent [installed in the cluster](install/index.md#install-the-agent-into-the-cluster). ## Use the CI/CD Tunnel to run Kubernetes commands from GitLab CI/CD @@ -37,6 +35,16 @@ there isn't any context selected. Contexts are named in the following format: `<agent-configuration-project-path>:<agent-name>`. To get the list of available contexts, run `kubectl config get-contexts`. +## Share the CI/CD Tunnel provided by an Agent with other projects and groups + +The Agent can be configured to enable access to the CI/CD Tunnel to other projects or all the projects under a given group. This way you can have a single agent serving all the requests for several projects saving on resources and maintenance. + +You can read more on how to [authorize access in the Agent configuration reference](repository.md#authorize-projects-and-groups-to-use-an-agent). + +## Restrict access of authorized projects and groups **(PREMIUM)** + +You can [configure various impersonations](repository.md#use-impersonation-to-restrict-project-and-group-access) to restrict the permissions of a shared CI/CD Tunnel. + ## Example for a `kubectl` command using the CI/CD Tunnel The following example shows a CI/CD job that runs a `kubectl` command using the CI/CD Tunnel. diff --git a/doc/user/clusters/agent/index.md b/doc/user/clusters/agent/index.md index 557d389147d..80b9f3f17f5 100644 --- a/doc/user/clusters/agent/index.md +++ b/doc/user/clusters/agent/index.md @@ -4,31 +4,58 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab Kubernetes Agent **(PREMIUM)** +# GitLab Kubernetes Agent **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/223061) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. -> - [Introduced](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/7) in GitLab 13.6, `grpcs` is supported. +> - Support for `grpcs` [introduced](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/issues/7) in GitLab 13.6. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300960) in GitLab 13.10, KAS became available on GitLab.com under `wss://kas.gitlab.com` through an Early Adopter Program. > - Introduced in GitLab 13.11, the GitLab Kubernetes Agent became available to every project on GitLab.com. +> - The GitLab Kubernetes Agent was [moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) to GitLab Free in 14.5. -The [GitLab Kubernetes Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) -is an active in-cluster component for solving GitLab and Kubernetes integration -tasks in a secure and cloud-native way. It enables: +The [GitLab Kubernetes Agent](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent) ("Agent", for short) +is an active in-cluster component for connecting Kubernetes clusters to GitLab safely to support cloud-native deployment, management, and monitoring. -- GitLab integration with a Kubernetes cluster behind a firewall or NAT - (network address translation). -- Pull-based GitOps deployments. -- [Inventory object](../../infrastructure/clusters/deploy/inventory_object.md) to keep track of objects applied to your cluster. -- Real-time access to API endpoints in a cluster. -- Alert generation based on [Container network policy](../../application_security/policies/index.md#container-network-policy). -- [CI/CD Tunnel](ci_cd_tunnel.md) that enables users to access Kubernetes clusters from GitLab CI/CD jobs even if there is no network connectivity between GitLab Runner and a cluster. +The Agent is installed into the cluster through code, providing you with a fast, safe, stable, and scalable solution. -Many more features are planned. Please review [our roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329) -and [our development documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/doc). +With GitOps, you can manage containerized clusters and applications from a Git repository that: -## GitLab Agent GitOps workflow +- Is the single source of truth of your system. +- Is the single place where you operate your system. +- Is a single resource to monitor your system. -The GitLab Agent, herein _Agent_, uses multiple GitLab projects to provide a flexible workflow +By combining GitLab, Kubernetes, and GitOps, it results in a robust infrastructure: + +- GitLab as the GitOps operator. +- Kubernetes as the automation and convergence system. +- GitLab CI/CD as the Continuous Integration and Continuous Deployment engine. + +Beyond that, you can use all the features offered by GitLab as +the all-in-one DevOps platform for your product and your team. + +## Agent's features + +By using the GitLab Kubernetes Agent, you can: + +- Connect GitLab with a Kubernetes cluster behind a firewall or a +Network Address Translation (NAT). +- Have real-time access to API endpoints in your cluster from GitLab CI/CD. +- Use GitOps to configure your cluster through the [Agent's repository](repository.md). +- Perform pull-based or push-based GitOps deployments. +- Configure [Network Security Alerts](#kubernetes-network-security-alerts) +based on [Container Network Policies](../../application_security/policies/index.md#container-network-policy). +- Track objects applied to your cluster through [inventory objects](../../infrastructure/clusters/deploy/inventory_object.md). +- Use the [CI/CD Tunnel](ci_cd_tunnel.md) to access Kubernetes clusters +from GitLab CI/CD jobs while keeping the cluster's APIs safe and unexposed +to the internet. +- [Deploy the GitLab Runner in a Kubernetes cluster](https://docs.gitlab.com/runner/install/kubernetes-agent.html). + +See the [GitLab Kubernetes Agent roadmap](https://gitlab.com/groups/gitlab-org/-/epics/3329) to track its development. + +To contribute to the Agent, see the [Agent's development documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/doc). + +## Agent's GitOps workflow **(PREMIUM)** + +The Agent uses multiple GitLab projects to provide a flexible workflow that can suit various needs. This diagram shows these repositories and the main actors involved in a deployment: @@ -50,365 +77,35 @@ sequenceDiagram end ``` -There are several components that work in concert for the Agent to accomplish GitOps deployments: - -- A properly-configured Kubernetes cluster where the Agent is running. -- A configuration repository that contains a `config.yaml` file, which tells the - Agent the repositories to synchronize with the cluster. -- A manifest repository that contains manifest files. Any changes to manifest files are applied to the cluster. - -You can use the same GitLab project or projects for configuration and manifest files, as follows: - -- Single GitLab project (recommended): when you use a single repository to hold both the manifest and the configuration files, these projects can be either private or public, as you prefer. -- Two GitLab projects: when you opt to use two different GitLab projects, one for manifest files, and another for configuration files, the manifests project must be public, while the configuration project can be either private or public. Our backlog contains issues for adding support for -[private manifest repositories outside of the configuration project](https://gitlab.com/gitlab-org/gitlab/-/issues/220912) and -[group level agents](https://gitlab.com/gitlab-org/gitlab/-/issues/283885) in the future. - -For more details, please refer to our [full architecture documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md#high-level-architecture) in the Agent project. - -## Get started with GitOps and the GitLab Agent - -The setup process involves a few steps to enable GitOps deployments: - -1. [Set up the Kubernetes Agent Server](#set-up-the-kubernetes-agent-server) for your GitLab instance. -1. [Define a configuration repository](#define-a-configuration-repository). -1. [Create an Agent record in GitLab](#create-an-agent-record-in-gitlab). -1. [Install the Agent into the cluster](#install-the-agent-into-the-cluster). -1. [Generate and copy a Secret token used to connect to the Agent](#create-the-kubernetes-secret). -1. [Create manifest files](#create-manifest-files). - -<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch a GitLab 14.2 [walking-through video](https://www.youtube.com/watch?v=XuBpKtsgGkE) with this process. - -### Upgrades and version compatibility - -As the GitLab Kubernetes Agent is a new product, we are constantly adding new features -to it. As a result, while shipped features are production ready, its internal API is -neither stable nor versioned yet. For this reason, GitLab only guarantees compatibility -between corresponding major.minor (X.Y) versions of GitLab and its cluster side -component, `agentk`. - -Upgrade your agent installations together with GitLab upgrades. To decide which version of `agentk` to install follow: - -1. Open the [`GITLAB_KAS_VERSION`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_KAS_VERSION) file from the GitLab Repository, which contains the latest `agentk` version associated with the `master` branch. -1. Change the `master` branch and select the Git tag associated with your version. For instance, you could change it to GitLab [v13.5.3-ee release](https://gitlab.com/gitlab-org/gitlab/-/blob/v13.5.3-ee/GITLAB_KAS_VERSION) - -The available `agentk` and `kas` versions can be found in -[the container registry](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/container_registry/). - -### Set up the Kubernetes Agent Server - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.10, the GitLab Kubernetes Agent Server (KAS) became available on GitLab.com under `wss://kas.gitlab.com`. - -To use the KAS: - -- If you are a self-managed user, follow the instructions to [install the Kubernetes Agent Server](../../../administration/clusters/kas.md). -- If you are a GitLab.com user, when you [set up the configuration repository](#define-a-configuration-repository) for your agent, use `wss://kas.gitlab.com` as the `--kas-address`. - -### Define a configuration repository - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7, the Agent manifest configuration can be added to multiple directories (or subdirectories) of its repository. -> - Group authorization was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. - -To configure an Agent, you need: - -1. A GitLab repository to hold the configuration file. -1. Install the Agent in a cluster. - -After installed, when you update the configuration file, GitLab transmits the -information to the cluster automatically without downtime. - -In your repository, add the Agent configuration file under: - -```plaintext -.gitlab/agents/<agent-name>/config.yaml -``` - -Your `config.yaml` file specifies all configurations of the Agent, such as: - -- The manifest projects to synchronize. -- The groups that can access this Agent via the [CI/CD Tunnel](ci_cd_tunnel.md). -- The address of the `hubble-relay` for the Network Security policy integrations. - -As an example, a minimal Agent configuration that sets up only the manifest -synchronizations is: - -```yaml -gitops: - manifest_projects: - - id: "path-to/your-manifest-project-1" - paths: - - glob: '/**/*.{yaml,yml,json}' -``` - -All the options for the [Kubernetes Agent configuration repository](repository.md) are documented separately. - -### Create an Agent record in GitLab - -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5786) in GitLab 14.1, you can create a new Agent record directly from the GitLab UI. - -Next, create a GitLab Rails Agent record to associate it with -the configuration repository project. Creating this record also creates a Secret needed to configure -the Agent in subsequent steps. - -In GitLab: - -1. From your project's sidebar, select **Infrastructure > Kubernetes clusters**. -1. Select the **GitLab Agent managed clusters** tab. -1. Select **Integrate with the GitLab Agent**. -1. From the **Select an Agent** dropdown menu, select the Agent you want to connect and select **Next** to access the installation form. -1. The form reveals your registration token. Securely store this secret token as you cannot view it again. -1. Copy the command under **Recommended installation method**. - -In your computer: - -1. Open your local terminal and connect to your cluster. -1. Run the command you copied from the installation form. - -### Install the Agent into the cluster - -To install the in-cluster component of the Agent, first you need to define a namespace. To create a new namespace, -for example, `gitlab-kubernetes-agent`, run: - -```shell -kubectl create namespace gitlab-kubernetes-agent -``` - -To perform a one-liner installation, run the command below. Make sure to replace: - -- `your-agent-token` with the token received from the previous step (identified as `secret` in the JSON output). -- `gitlab-kubernetes-agent` with the namespace you defined in the previous step. -- `wss://kas.gitlab.example.com` with the configured access of the Kubernetes Agent Server (KAS). For GitLab.com users, the KAS is available under `wss://kas.gitlab.com`. -- `--agent-version=vX.Y.Z` with the latest released patch version matching your GitLab installation's major and minor versions. For example, for GitLab v13.9.0, use `--agent-version=v13.9.1`. You can find your GitLab version under the "Help/Help" menu. - -```shell -docker run --pull=always --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate --agent-token=your-agent-token --kas-address=wss://kas.gitlab.example.com --agent-version=vX.Y.Z --namespace gitlab-kubernetes-agent | kubectl apply -f - -``` - -WARNING: -`--agent-version stable` can be used to refer to the latest stable release at the time when the command runs. It's fine for -testing purposes but for production please make sure to specify a matching version explicitly. - -To find out the various options the above Docker container supports, run: - -```shell -docker run --pull=always --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate --help -``` - -#### Advanced installation - -For more advanced configurations, we recommend to use [the `kpt` based installation method](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/build/deployment/gitlab-agent). - -Otherwise, follow the manual installation steps described below. - -### Create the Kubernetes secret - -After generating the token, you must apply it to the Kubernetes cluster. - -To create your Secret, run: - -```shell -kubectl create secret generic -n gitlab-kubernetes-agent gitlab-kubernetes-agent-token --from-literal=token='YOUR_AGENT_TOKEN' -``` - -The following example file contains the -Kubernetes resources required for the Agent to be installed. You can modify this -example [`resources.yml` file](#example-resourcesyml-file) in the following ways: - -- Replace `namespace: gitlab-kubernetes-agent` with `namespace: <YOUR-DESIRED-NAMESPACE>`. -- You can configure `kas-address` (Kubernetes Agent Server) in several ways. - The agent can use the WebSockets or gRPC protocols to connect to the Agent Server. - Select the option appropriate for your cluster configuration and GitLab architecture: - - The `wss` scheme (an encrypted WebSockets connection) is specified by default - after you install the `gitlab-kas` sub-chart, or enable `gitlab-kas` for Omnibus GitLab. - When using the sub-chart, you must set `wss://kas.host.tld:443` as - `kas-address`, where `host.tld` is the domain you've setup for your GitLab installation. - When using Omnibus GitLab, you must set `wss://GitLab.host.tld:443/-/kubernetes-agent/` as - `kas-address`, where `GitLab.host.tld` is your GitLab hostname. - - When using the sub-chart, specify the `ws` scheme (such as `ws://kas.host.tld:80`) - to use an unencrypted WebSockets connection. - When using the Omnibus GitLab, specify the `ws` scheme (such as `ws://GitLab.host.tld:80/-/kubernetes-agent/`). - - Specify the `grpc` scheme if both Agent and Server are installed in one cluster. - In this case, you may specify `kas-address` value as - `grpc://gitlab-kas.<your-namespace>:8150`) to use gRPC directly, where `gitlab-kas` - is the name of the service created by `gitlab-kas` chart, and `<your-namespace>` - is the namespace where the chart was installed. - - Specify the `grpcs` scheme to use an encrypted gRPC connection. - - When deploying KAS through the [GitLab chart](https://docs.gitlab.com/charts/), it's possible to customize the - `kas-address` for `wss` and `ws` schemes to whatever you need. - Check the [chart's KAS Ingress documentation](https://docs.gitlab.com/charts/charts/gitlab/kas/#ingress) - to learn more about it. - - In the near future, Omnibus GitLab intends to provision `gitlab-kas` under a sub-domain by default, instead of the `/-/kubernetes-agent/` path. Please follow [this issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5784) for details. -- If you defined your own secret name, replace `gitlab-kubernetes-agent-token` with your - secret name in the `secretName:` section. - -To apply this file, run the following command: - -```shell -kubectl apply -n gitlab-kubernetes-agent -f ./resources.yml -``` - -To review your configuration, run the following command: - -```shell -$ kubectl get pods -n gitlab-kubernetes-agent - -NAMESPACE NAME READY STATUS RESTARTS AGE -gitlab-kubernetes-agent gitlab-kubernetes-agent-77689f7dcb-5skqk 1/1 Running 0 51s -``` - -#### Example `resources.yml` file - -```yaml ---- -apiVersion: v1 -kind: Namespace -metadata: - name: gitlab-kubernetes-agent ---- -apiVersion: v1 -kind: ServiceAccount -metadata: - name: gitlab-kubernetes-agent ---- -apiVersion: apps/v1 -kind: Deployment -metadata: - name: gitlab-kubernetes-agent -spec: - replicas: 1 - selector: - matchLabels: - app: gitlab-kubernetes-agent - template: - metadata: - labels: - app: gitlab-kubernetes-agent - spec: - serviceAccountName: gitlab-kubernetes-agent - containers: - - name: agent - # Make sure to specify a matching version for production - image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:vX.Y.Z - args: - - --token-file=/config/token - - --kas-address - - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com. - # - wss://gitlab.host.tld:443/-/kubernetes-agent/ - # - wss://kas.gitlab.com # for GitLab.com users, use this KAS. - # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker. - volumeMounts: - - name: token-volume - mountPath: /config - volumes: - - name: token-volume - secret: - secretName: gitlab-kubernetes-agent-token - strategy: - type: RollingUpdate - rollingUpdate: - maxSurge: 0 - maxUnavailable: 1 ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: gitlab-kubernetes-agent-write -rules: -- resources: - - '*' - apiGroups: - - '*' - verbs: - - create - - update - - delete - - patch ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: gitlab-kubernetes-agent-write-binding -roleRef: - name: gitlab-kubernetes-agent-write - kind: ClusterRole - apiGroup: rbac.authorization.k8s.io -subjects: -- name: gitlab-kubernetes-agent - kind: ServiceAccount - namespace: gitlab-kubernetes-agent ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRole -metadata: - name: gitlab-kubernetes-agent-read -rules: -- resources: - - '*' - apiGroups: - - '*' - verbs: - - get - - list - - watch ---- -apiVersion: rbac.authorization.k8s.io/v1 -kind: ClusterRoleBinding -metadata: - name: gitlab-kubernetes-agent-read-binding -roleRef: - name: gitlab-kubernetes-agent-read - kind: ClusterRole - apiGroup: rbac.authorization.k8s.io -subjects: -- name: gitlab-kubernetes-agent - kind: ServiceAccount - namespace: gitlab-kubernetes-agent -``` - -### Create manifest files - -In a previous step, you configured a `config.yaml` to point to the GitLab projects -the Agent should synchronize. Agent monitors each of those projects for changes to the manifest files it contains. You can auto-generate manifest files with a -templating engine or other means. - -The agent is authorized to download manifests for the configuration -project, and public projects. Support for other private projects is -planned in the issue [Agent authorization for private manifest -projects](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). - -Each time you push a change to a monitored manifest repository, the Agent logs the change: - -```plaintext -2020-09-15_14:09:04.87946 gitlab-k8s-agent : time="2020-09-15T10:09:04-04:00" level=info msg="Config: new commit" agent_id=1 commit_id=e6a3651f1faa2e928fe6120e254c122451be4eea -``` +For more details, refer to our [architecture documentation](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/architecture.md#high-level-architecture) in the Agent project. -#### Example manifest file +## Install the Agent in your cluster -This file creates a minimal `ConfigMap`: +See how to [install the GitLab Kubernetes Agent in your cluster](install/index.md). -```yaml -apiVersion: v1 -kind: ConfigMap -metadata: - name: demo-map - namespace: gitlab-kubernetes-agent # Can be any namespace managed by you that the agent has access to. -data: - key: value -``` +## GitOps deployments **(PREMIUM)** -## Example projects +To perform GitOps deployments with the Agent, you need: -The following example projects can help you get started with the Kubernetes Agent. +- A properly-configured Kubernetes cluster where the Agent is running. +- A [configuration repository](repository.md) that contains a +`config.yaml` file, which tells the Agent the repositories to synchronize +with the cluster. +- A manifest repository that contains manifest files. Any changes to manifest files are applied to the cluster. -- [Configuration repository](https://gitlab.com/gitlab-org/configure/examples/kubernetes-agent) -- This basic GitOps example deploys NGINX: [Manifest repository](https://gitlab.com/gitlab-org/configure/examples/gitops-project) +You can use a single GitLab project or different projects for the Agent +configuration and manifest files, as follows: -### GitLab Runner Deployment with the Agent +- Single GitLab project (recommended): When you use a single repository to hold + both the manifest and the configuration files, these projects can be either + private or public. +- Two GitLab projects: When you use two different GitLab projects (one for + manifest files and another for configuration files), the manifests project must + be public, while the configuration project can be either private or public. -You can use the Kubernetes Agent to -[deploy GitLab Runner in a Kubernetes cluster](https://docs.gitlab.com/runner/install/kubernetes-agent.html). +Support for separated private manifest and configuration repositories is tracked in this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). -## Kubernetes Network Security Alerts +## Kubernetes Network Security Alerts **(ULTIMATE)** The GitLab Agent also provides an integration with Cilium. This integration provides a simple way to generate network policy-related alerts and to surface those alerts in GitLab. @@ -426,24 +123,12 @@ There are several components that work in concert for the Agent to generate the - Add the required labels and annotations to existing network policies. - A configuration repository with [Cilium configured in `config.yaml`](repository.md#surface-network-security-alerts-from-cluster-to-gitlab) -The setup process follows the same steps as [GitOps](#get-started-with-gitops-and-the-gitlab-agent), +The setup process follows the same [Agent's installation steps](install/index.md), with the following differences: - When you define a configuration repository, you must do so with [Cilium settings](repository.md#surface-network-security-alerts-from-cluster-to-gitlab). - You do not need to specify the `gitops` configuration section. -## Management interfaces - -Users with at least the [Developer](../../permissions.md) can access the user interface -for the GitLab Kubernetes agent at **Infrastructure > Kubernetes clusters**, under the -**GitLab Agent managed clusters** tab. This page lists all registered agents for -the current project, and the configuration directory for each agent: - - - -Additional management interfaces are planned for the GitLab Kubernetes Agent. -[Provide more feedback in the related epic](https://gitlab.com/groups/gitlab-org/-/epics/4739). - ## Remove the GitLab Kubernetes Agent 1. Remove an Agent record with GraphQL by deleting the `clusterAgent` and the `clusterAgentToken`. @@ -518,7 +203,7 @@ specified the `kas-address` correctly. ``` This error occurs if the `kas-address` doesn't include a trailing slash. To fix it, make sure that the -`wss` or `ws` URL ends with a training slash, such as `wss://GitLab.host.tld:443/-/kubernetes-agent/` +`wss` or `ws` URL ends with a trailing slash, such as `wss://GitLab.host.tld:443/-/kubernetes-agent/` or `ws://GitLab.host.tld:80/-/kubernetes-agent/`. #### ValidationError(Deployment.metadata) diff --git a/doc/user/clusters/agent/install/index.md b/doc/user/clusters/agent/install/index.md new file mode 100644 index 00000000000..fad9d4f08f1 --- /dev/null +++ b/doc/user/clusters/agent/install/index.md @@ -0,0 +1,369 @@ +--- +stage: Configure +group: Configure +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Install the GitLab Kubernetes Agent **(FREE)** + +> [Moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) to GitLab Free in 14.5. + +To get started with the GitLab Kubernetes Agent, install it in your cluster. + +Pre-requisites: + +- An existing Kubernetes cluster. +- An account on GitLab. + +## Installation steps + +To install the [GitLab Kubernetes Agent](../index.md) in your cluster: + +1. [Set up the Kubernetes Agent Server](#set-up-the-kubernetes-agent-server) for your GitLab instance. +1. [Define a configuration repository](#define-a-configuration-repository). +1. [Create an Agent record in GitLab](#create-an-agent-record-in-gitlab). +1. [Install the Agent into the cluster](#install-the-agent-into-the-cluster). +1. [Generate and copy a Secret token used to connect to the Agent](#create-the-kubernetes-secret). +1. [Create manifest files](#create-manifest-files). + +<i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch a GitLab 14.2 [walking-through video](https://www.youtube.com/watch?v=XuBpKtsgGkE) with this process. + +### Set up the Kubernetes Agent Server + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.10, the GitLab Kubernetes Agent Server (KAS) became available on GitLab.com under `wss://kas.gitlab.com`. + +To use the KAS: + +- If you are a self-managed user, follow the instructions to [install the Kubernetes Agent Server](../../../../administration/clusters/kas.md). +- If you are a GitLab.com user, when you [set up the configuration repository](#define-a-configuration-repository) for your agent, use `wss://kas.gitlab.com` as the `--kas-address`. + +### Define a configuration repository + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in GitLab 13.7, the Agent manifest configuration can be added to multiple directories (or subdirectories) of its repository. +> - Group authorization was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. + +To configure an Agent, you need: + +1. A GitLab repository to hold the configuration file. +1. Install the Agent in a cluster. + +After installed, when you update the configuration file, GitLab transmits the +information to the cluster automatically without downtime. + +In your repository, add the Agent configuration file under: + +```plaintext +.gitlab/agents/<agent-name>/config.yaml +``` + +Make sure that `<agent-name>` conforms to the [Agent's naming format](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md#agent-identity-and-name). + +Your `config.yaml` file specifies all configurations of the Agent, such as: + +- The manifest projects to synchronize. +- The groups that can access this Agent via the [CI/CD Tunnel](../ci_cd_tunnel.md). +- The address of the `hubble-relay` for the Network Security policy integrations. + +As an example, a minimal Agent configuration that sets up only the manifest +synchronizations is: + +```yaml +gitops: + manifest_projects: + # The `id` is the path to the Git repository holding your manifest files + - id: "path/to/your-manifest-project-1" + paths: + - glob: '/**/*.{yaml,yml,json}' +``` + +All the options for the [Kubernetes Agent configuration repository](../repository.md) are documented separately. + +### Create an Agent record in GitLab + +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5786) in GitLab 14.1, you can create a new Agent record directly from the GitLab UI. + +Next, create a GitLab Rails Agent record to associate it with +the configuration repository project. Creating this record also creates a Secret needed to configure +the Agent in subsequent steps. + +In GitLab: + +1. Ensure that [GitLab CI/CD is enabled in your project](../../../../ci/enable_or_disable_ci.md#enable-cicd-in-a-project). +1. From your project's sidebar, select **Infrastructure > Kubernetes clusters**. +1. Select **Actions**. +1. From the **Select an Agent** dropdown, select the Agent you want to connect and select **Register Agent** to access the installation form. +1. The form reveals your registration token. Securely store this secret token as you cannot view it again. +1. Copy the command under **Recommended installation method**. + +In your computer: + +1. Open your local terminal and connect to your cluster. +1. Run the command you copied from the installation form. + +### Install the Agent into the cluster + +To install the in-cluster component of the Agent, first you need to define a namespace. To create a new namespace, +for example, `gitlab-kubernetes-agent`, run: + +```shell +kubectl create namespace gitlab-kubernetes-agent +``` + +To perform a one-liner installation, run the command below. Make sure to replace: + +- `your-agent-token` with the token received from the previous step (identified as `secret` in the JSON output). +- `gitlab-kubernetes-agent` with the namespace you defined in the previous step. +- `wss://kas.gitlab.example.com` with the configured access of the Kubernetes Agent Server (KAS). For GitLab.com users, the KAS is available under `wss://kas.gitlab.com`. +- `--agent-version=vX.Y.Z` with the latest released patch version matching your GitLab installation's major and minor versions. For example, for GitLab v13.9.0, use `--agent-version=v13.9.1`. You can find your GitLab version under the "Help/Help" menu. + +```shell +docker run --pull=always --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate --agent-token=your-agent-token --kas-address=wss://kas.gitlab.example.com --agent-version=vX.Y.Z --namespace gitlab-kubernetes-agent | kubectl apply -f - +``` + +WARNING: +`--agent-version stable` can be used to refer to the latest stable release at the time when the command runs. It's fine for +testing purposes but for production please make sure to specify a matching version explicitly. + +To find out the various options the above Docker container supports, run: + +```shell +docker run --pull=always --rm registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/cli:stable generate --help +``` + +## Advanced installation + +For more advanced configurations, we recommend to use [the `kpt` based installation method](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/tree/master/build/deployment/gitlab-agent). + +Otherwise, follow the manual installation steps described below. + +### Create the Kubernetes secret + +After generating the token, you must apply it to the Kubernetes cluster. + +To create your Secret, run: + +```shell +kubectl create secret generic -n gitlab-kubernetes-agent gitlab-kubernetes-agent-token --from-literal=token='YOUR_AGENT_TOKEN' +``` + +The following example file contains the +Kubernetes resources required for the Agent to be installed. You can modify this +example [`resources.yml` file](#example-resourcesyml-file) in the following ways: + +- Replace `namespace: gitlab-kubernetes-agent` with `namespace: <YOUR-DESIRED-NAMESPACE>`. +- You can configure `kas-address` (Kubernetes Agent Server) in several ways. + The agent can use the WebSockets or gRPC protocols to connect to the Agent Server. + Select the option appropriate for your cluster configuration and GitLab architecture: + - The `wss` scheme (an encrypted WebSockets connection) is specified by default + after you install the `gitlab-kas` sub-chart, or enable `gitlab-kas` for Omnibus GitLab. + When using the sub-chart, you must set `wss://kas.host.tld:443` as + `kas-address`, where `host.tld` is the domain you've setup for your GitLab installation. + When using Omnibus GitLab, you must set `wss://GitLab.host.tld:443/-/kubernetes-agent/` as + `kas-address`, where `GitLab.host.tld` is your GitLab hostname. + - When using the sub-chart, specify the `ws` scheme (such as `ws://kas.host.tld:80`) + to use an unencrypted WebSockets connection. + When using the Omnibus GitLab, specify the `ws` scheme (such as `ws://GitLab.host.tld:80/-/kubernetes-agent/`). + - Specify the `grpc` scheme if both Agent and Server are installed in one cluster. + In this case, you may specify `kas-address` value as + `grpc://gitlab-kas.<your-namespace>:8150`) to use gRPC directly, where `gitlab-kas` + is the name of the service created by `gitlab-kas` chart, and `<your-namespace>` + is the namespace where the chart was installed. + - Specify the `grpcs` scheme to use an encrypted gRPC connection. + - When deploying KAS through the [GitLab chart](https://docs.gitlab.com/charts/), it's possible to customize the + `kas-address` for `wss` and `ws` schemes to whatever you need. + Check the [chart's KAS Ingress documentation](https://docs.gitlab.com/charts/charts/gitlab/kas/#ingress) + to learn more about it. + - In the near future, Omnibus GitLab intends to provision `gitlab-kas` under a sub-domain by default, instead of the `/-/kubernetes-agent/` path. Please follow [this issue](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5784) for details. +- If you defined your own secret name, replace `gitlab-kubernetes-agent-token` with your + secret name in the `secretName:` section. + +To apply this file, run the following command: + +```shell +kubectl apply -n gitlab-kubernetes-agent -f ./resources.yml +``` + +To review your configuration, run the following command: + +```shell +$ kubectl get pods -n gitlab-kubernetes-agent + +NAMESPACE NAME READY STATUS RESTARTS AGE +gitlab-kubernetes-agent gitlab-kubernetes-agent-77689f7dcb-5skqk 1/1 Running 0 51s +``` + +#### Example `resources.yml` file + +```yaml +--- +apiVersion: v1 +kind: Namespace +metadata: + name: gitlab-kubernetes-agent +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: gitlab-kubernetes-agent +--- +apiVersion: apps/v1 +kind: Deployment +metadata: + name: gitlab-kubernetes-agent +spec: + replicas: 1 + selector: + matchLabels: + app: gitlab-kubernetes-agent + template: + metadata: + labels: + app: gitlab-kubernetes-agent + spec: + serviceAccountName: gitlab-kubernetes-agent + containers: + - name: agent + # Make sure to specify a matching version for production + image: "registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:vX.Y.Z" + args: + - --token-file=/config/token + - --kas-address + - wss://kas.host.tld:443 # replace this line with the line below if using Omnibus GitLab or GitLab.com. + # - wss://gitlab.host.tld:443/-/kubernetes-agent/ + # - wss://kas.gitlab.com # for GitLab.com users, use this KAS. + # - grpc://host.docker.internal:8150 # use this attribute when connecting from Docker. + volumeMounts: + - name: token-volume + mountPath: /config + volumes: + - name: token-volume + secret: + secretName: gitlab-kubernetes-agent-token + strategy: + type: RollingUpdate + rollingUpdate: + maxSurge: 0 + maxUnavailable: 1 +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: gitlab-kubernetes-agent-write +rules: +- resources: + - '*' + apiGroups: + - '*' + verbs: + - create + - update + - delete + - patch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: gitlab-kubernetes-agent-write-binding +roleRef: + name: gitlab-kubernetes-agent-write + kind: ClusterRole + apiGroup: rbac.authorization.k8s.io +subjects: +- name: gitlab-kubernetes-agent + kind: ServiceAccount + namespace: gitlab-kubernetes-agent +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRole +metadata: + name: gitlab-kubernetes-agent-read +rules: +- resources: + - '*' + apiGroups: + - '*' + verbs: + - get + - list + - watch +--- +apiVersion: rbac.authorization.k8s.io/v1 +kind: ClusterRoleBinding +metadata: + name: gitlab-kubernetes-agent-read-binding +roleRef: + name: gitlab-kubernetes-agent-read + kind: ClusterRole + apiGroup: rbac.authorization.k8s.io +subjects: +- name: gitlab-kubernetes-agent + kind: ServiceAccount + namespace: gitlab-kubernetes-agent +``` + +### Create manifest files + +In a previous step, you configured a `config.yaml` to point to the GitLab projects +the Agent should synchronize. Agent monitors each of those projects for changes to the manifest files it contains. You can auto-generate manifest files with a +templating engine or other means. + +The agent is authorized to download manifests for the configuration +project, and public projects. Support for other private projects is +planned in the issue [Agent authorization for private manifest +projects](https://gitlab.com/gitlab-org/gitlab/-/issues/220912). + +Each time you push a change to a monitored manifest repository, the Agent logs the change: + +```plaintext +2020-09-15_14:09:04.87946 gitlab-k8s-agent : time="2020-09-15T10:09:04-04:00" level=info msg="Config: new commit" agent_id=1 commit_id=e6a3651f1faa2e928fe6120e254c122451be4eea +``` + +#### Example manifest file + +This file creates a minimal `ConfigMap`: + +```yaml +apiVersion: v1 +kind: ConfigMap +metadata: + name: demo-map + namespace: gitlab-kubernetes-agent # Can be any namespace managed by you that the agent has access to. +data: + key: value +``` + +## Example projects + +The following example projects can help you get started with the Kubernetes Agent. + +- [Configuration repository](https://gitlab.com/gitlab-org/configure/examples/kubernetes-agent) +- This basic GitOps example deploys NGINX: [Manifest repository](https://gitlab.com/gitlab-org/configure/examples/gitops-project) + +## View installed Agents + +Users with at least the [Developer](../../../permissions.md) can access the user interface +for the GitLab Kubernetes Agent at **Infrastructure > Kubernetes clusters**, under the +**Agent** tab. This page lists all registered agents for the current project, +and the configuration directory for each agent: + + + +Additional management interfaces are planned for the GitLab Kubernetes Agent. +[Provide more feedback in the related epic](https://gitlab.com/groups/gitlab-org/-/epics/4739). + +## Upgrades and version compatibility + +The GitLab Kubernetes Agent is comprised of two major components: `agentk` and `kas`. +As we provide `kas` installers built into the various GitLab installation methods, the required `kas` version corresponds to the GitLab `major.minor` (X.Y) versions. + +At the same time, `agentk` and `kas` can differ by 1 minor version in either direction. For example, +`agentk` 14.4 supports `kas` 14.3, 14.4, and 14.5 (regardless of the patch). + +A feature introduced in a given GitLab minor version might work with other `agentk` or `kas` versions. +To make sure that it works, use at least the same `agentk` and `kas` minor version. For example, +if your GitLab version is 14.2, use at least `agentk` 14.2 and `kas` 14.2. + +We recommend upgrading your `kas` installations together with GitLab instances' upgrades, and to upgrade the `agentk` installations after upgrading GitLab. + +The available `agentk` and `kas` versions can be found in +[the container registry](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/container_registry/). diff --git a/doc/user/clusters/agent/repository.md b/doc/user/clusters/agent/repository.md index cbb27a3f343..6ceb2766cc9 100644 --- a/doc/user/clusters/agent/repository.md +++ b/doc/user/clusters/agent/repository.md @@ -4,12 +4,13 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#designated-technical-writers --- -# Kubernetes Agent configuration repository **(PREMIUM)** +# Kubernetes Agent configuration repository **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/259669) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.7. > - [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/3834) in GitLab 13.11, the Kubernetes Agent became available on GitLab.com. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/332227) in GitLab 14.0, the `resource_inclusions` and `resource_exclusions` attributes were removed and `reconcile_timeout`, `dry_run_strategy`, `prune`, `prune_timeout`, `prune_propagation_policy`, and `inventory_policy` attributes were added. > - The `ci_access` attribute was [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. +> - The GitLab Kubernetes Agent was [moved](https://gitlab.com/groups/gitlab-org/-/epics/6290) to GitLab Free in 14.5. WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -22,17 +23,19 @@ The Agent bootstraps with the GitLab installation URL and an authentication toke and you provide the rest of the configuration in your repository, following Infrastructure as Code (IaaC) best practices. -A minimal repository layout looks like this, with `my_agent_1` as the name +A minimal repository layout looks like this, with `my-agent-1` as the name of your Agent: ```plaintext |- .gitlab |- agents - |- my_agent_1 + |- my-agent-1 |- config.yaml ``` -## Synchronize manifest projects +Make sure that `<agent-name>` conforms to the [Agent's naming format](https://gitlab.com/gitlab-org/cluster-integration/gitlab-agent/-/blob/master/doc/identity_and_auth.md#agent-identity-and-name). + +## Synchronize manifest projects **(PREMIUM)** Your `config.yaml` file contains a `gitops` section, which contains a `manifest_projects` section. Each `id` in the `manifest_projects` section is the path to a Git repository @@ -132,7 +135,7 @@ INCORRECT - both globs match `*.yaml` files in the root directory: ```yaml gitops: manifest_projects: - - id: project1 + - id: project1 paths: - glob: '/**/*.yaml' - glob: '/*.yaml' @@ -143,51 +146,140 @@ CORRECT - single globs matches all `*.yaml` files recursively: ```yaml gitops: manifest_projects: - - id: project1 + - id: project1 paths: - glob: '/**/*.yaml' ``` -## Authorize groups to use an Agent +## Authorize projects and groups to use an Agent + +> - Group authorization [introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. +> - Project authorization [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/327850) in GitLab 14.4. + +If you use the same cluster across multiple projects, you can set up the [CI/CD Tunnel](ci_cd_tunnel.md) +to grant access to an Agent from one or more projects or groups. This way, all the authorized +projects can access the same Agent, which facilitates you to save resources and have a scalable setup. + +When you authorize a project to use an agent through the [CI/CD Tunnel](ci_cd_tunnel.md), +the selected Kubernetes context is automatically injected into CI/CD jobs, allowing you to +run Kubernetes commands from your authorized projects' scripts. When you authorize a group, +all the projects that belong to that group can access the selected agent. -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/5784) in GitLab 14.3. +An Agent can only authorize projects or groups in the same group hierarchy as the Agent's configuration +project. You can authorize up to 100 projects and 100 groups per Agent. -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -ask an administrator to [enable the `group_authorized_agents` flag](../../../administration/feature_flags.md). -On GitLab.com, this feature is available. +### Authorize projects to use an Agent -If you use the same cluster across multiple projects, you can set up the CI/CD Tunnel -to grant the Agent access to one or more groups. This way, all the projects that belong -to the authorized groups can access the same Agent. This enables you to save resources and -have a scalable setup. +To grant projects access to the Agent through the [CI/CD Tunnel](ci_cd_tunnel.md): -When you authorize a group, the agent's Kubernetes context is automatically injected -into every project of the authorized group, and users can select the connection as -described in the [CI/CD Tunnel documentation](ci_cd_tunnel.md). -To authorize a group to access the Agent through the [CI/CD Tunnel](ci_cd_tunnel.md), -use the `ci_access` attribute in your `config.yaml` configuration file. +1. Go to your Agent's configuration project. +1. Edit the Agent's configuration file (`config.yaml`). +1. Add the `projects` attribute into `ci_access`. +1. Identify the project through its path: -An Agent can only authorize groups in the same group hierarchy as the Agent's configuration project. At most -100 groups can be authorized per Agent. + ```yaml + ci_access: + projects: + - id: path/to/project + ``` -To authorize a group: +### Authorize groups to use an Agent -1. Edit your `config.yaml` file under the `.gitlab/agents/<agent name>` directory. -1. Add the `ci_access` root attribute. +To grant access to all projects within a group: + +1. Go to your Agent's configuration project. +1. Edit the Agent's configuration file (`config.yaml`). 1. Add the `groups` attribute into `ci_access`. -1. Add the group `id` into `groups`, identifying the authorized group through its path. +1. Identify the group or subgroup through its path: + + ```yaml + ci_access: + groups: + - id: path/to/group/subgroup + ``` + +### Use impersonation to restrict project and group access **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/345014) in GitLab 14.5. + +By default, the [CI/CD Tunnel](ci_cd_tunnel.md) inherits all the permissions from the service account used to install the +Agent in the cluster. +To restrict access to your cluster, you can use [impersonation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation). + +To specify impersonations, use the `access_as` attribute in your Agent's configuration file and use Kubernetes RBAC rules to manage impersonated account permissions. + +You can impersonate: + +- The Agent itself (default). +- The CI job that accesses the cluster. +- A specific user or system account defined within the cluster. + +#### Impersonate the Agent -For example: +The Agent is impersonated by default. You don't need to do anything to impersonate it. + +#### Impersonate the CI job that accesses the cluster + +To impersonate the CI job that accesses the cluster, add the `ci_job: {}` key-value +under the `access_as` key. + +When the agent makes the request to the actual Kubernetes API, it sets the +impersonation credentials in the following way: + +- `UserName` is set to `gitlab:ci_job:<job id>`. Example: `gitlab:ci_job:1074499489`. +- `Groups` is set to: + - `gitlab:ci_job` to identify all requests coming from CI jobs. + - The list of IDs of groups the project is in. + - The project ID. + - The slug of the environment this job belongs to. + + Example: for a CI job in `group1/group1-1/project1` where: + + - Group `group1` has ID 23. + - Group `group1/group1-1` has ID 25. + - Project `group1/group1-1/project1` has ID 150. + - Job running in a prod environment. + + Group list would be `[gitlab:ci_job, gitlab:group:23, gitlab:group:25, gitlab:project:150, gitlab:project_env:150:prod]`. + +- `Extra` carries extra information about the request. The following properties are set on the impersonated identity: + +| Property | Description | +| -------- | ----------- | +| `agent.gitlab.com/id` | Contains the agent ID. | +| `agent.gitlab.com/config_project_id` | Contains the agent's configuration project ID. | +| `agent.gitlab.com/project_id` | Contains the CI project ID. | +| `agent.gitlab.com/ci_pipeline_id` | Contains the CI pipeline ID. | +| `agent.gitlab.com/ci_job_id` | Contains the CI job ID. | +| `agent.gitlab.com/username` | Contains the username of the user the CI job is running as. | +| `agent.gitlab.com/environment_slug` | Contains the slug of the environment. Only set if running in an environment. | + +Example to restrict access by the CI job's identity: ```yaml ci_access: - # This agent is accessible from CI jobs in projects in these groups - groups: - - id: group/subgroup + projects: + - id: path/to/project + access_as: + ci_job: {} ``` -## Surface network security alerts from cluster to GitLab +#### Impersonate a static identity + +For the given CI/CD Tunnel connection, you can use a static identity for the impersonation. + +Add the `impersonate` key under the `access_as` key to make the request using the provided identity. + +The identity can be specified with the following keys: + +- `username` (required) +- `uid` +- `groups` +- `extra` + +See the [official Kubernetes documentation for more details](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation) on the usage of these keys. + +## Surface network security alerts from cluster to GitLab **(ULTIMATE)** The GitLab Agent provides an [integration with Cilium](index.md#kubernetes-network-security-alerts). To integrate, add a top-level `cilium` section to your `config.yml` file. Currently, the @@ -207,6 +299,90 @@ cilium: hubble_relay_address: "hubble-relay.gitlab-managed-apps.svc.cluster.local:80" ``` +## Scan your container images for vulnerabilities + +You can use [cluster image scanning](../../application_security/cluster_image_scanning/index.md) +to scan container images in your cluster for security vulnerabilities. + +To begin scanning all resources in your cluster, add a `starboard` +configuration block to your agent's `config.yaml` with no `filters`: + +```yaml +starboard: + vulnerability_report: + filters: [] +``` + +The namespaces that are able to be scanned depend on the [Starboard Operator install mode](https://aquasecurity.github.io/starboard/latest/operator/configuration/#install-modes). +By default, the Starboard Operator only scans resources in the `default` namespace. To change this +behavior, edit the `STARBOARD_OPERATOR` environment variable in the `starboard-operator` deployment +definition. + +By adding filters, you can limit scans by: + +- Resource name +- Kind +- Container name +- Namespace + +```yaml +starboard: + vulnerability_report: + filters: + - namespaces: + - staging + - production + kinds: + - Deployment + - DaemonSet + containers: + - ruby + - postgres + - nginx + resources: + - my-app-name + - postgres + - ingress-nginx +``` + +A resource is scanned if the resource matches any of the given names and all of the given filter +types (`namespaces`, `kinds`, `containers`, `resources`). If a filter type is omitted, then all +names are scanned. In this example, a resource isn't scanned unless it has a container named `ruby`, +`postgres`, or `nginx`, and it's a `Deployment`: + +```yaml +starboard: + vulnerability_report: + filters: + - kinds: + - Deployment + containers: + - ruby + - postgres + - nginx +``` + +There is also a global `namespaces` field that applies to all filters: + +```yaml +starboard: + vulnerability_report: + namespaces: + - production + filters: + - kinds: + - Deployment + - kinds: + - DaemonSet + resources: + - log-collector +``` + +In this example, the following resources are scanned: + +- All deployments (`Deployment`) in the `production` namespace +- All daemon sets (`DaemonSet`) named `log-collector` in the `production` namespace + ## Debugging To debug the cluster-side component (`agentk`) of the GitLab Kubernetes Agent, set the log diff --git a/doc/user/clusters/applications.md b/doc/user/clusters/applications.md index 31dd503a0cf..88382648b04 100644 --- a/doc/user/clusters/applications.md +++ b/doc/user/clusters/applications.md @@ -20,7 +20,7 @@ methods: "one-click install" and "CI/CD template". Both methods are **deprecated Both methods were limiting as you couldn't fully customize your third-party apps installed through GitLab Managed Apps. Therefore, we decided to deprecate this feature and provide -better [GitOps-driven alternatives](https://about.gitlab.com/direction/configure/kubernetes_management/#gitlab-managed-applications) to our users, such as [cluster integrations](integrations.md#cluster-integrations) and [cluster management project](management_project.md). +better [GitOps-driven alternatives](https://about.gitlab.com/direction/configure/kubernetes_management/#gitlab-managed-applications) to our users, such as [cluster integrations](integrations.md) and [cluster management project](management_project.md). ## Install using GitLab CI/CD (DEPRECATED) diff --git a/doc/user/clusters/cost_management.md b/doc/user/clusters/cost_management.md index 26611c26e5e..3ad994ecd7e 100644 --- a/doc/user/clusters/cost_management.md +++ b/doc/user/clusters/cost_management.md @@ -4,9 +4,13 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Cluster cost management **(ULTIMATE)** +# Cluster cost management (DEPRECATED) **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216737) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216737) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.5. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. Cluster cost management provides insights into cluster resource usage. GitLab provides an example [`kubecost-cost-model`](https://gitlab.com/gitlab-examples/kubecost-cost-model/) diff --git a/doc/user/clusters/crossplane.md b/doc/user/clusters/crossplane.md index 8906d1224b1..e6540e68f71 100644 --- a/doc/user/clusters/crossplane.md +++ b/doc/user/clusters/crossplane.md @@ -4,7 +4,12 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Crossplane configuration **(FREE)** +# Crossplane configuration (DEPRECATED) **(FREE)** + +> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. After [installing](applications.md#install-crossplane-using-gitlab-cicd) Crossplane, you must configure it for use. The process of configuring Crossplane includes: diff --git a/doc/user/clusters/environments.md b/doc/user/clusters/environments.md index 470f65db61b..72bc7b398dc 100644 --- a/doc/user/clusters/environments.md +++ b/doc/user/clusters/environments.md @@ -4,10 +4,14 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Cluster Environments **(PREMIUM)** +# Cluster Environments (DEPRECATED) **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13392) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3 for group-level clusters. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/14809) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.4 for instance-level clusters. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. Cluster environments provide a consolidated view of which CI [environments](../../ci/environments/index.md) are deployed to the Kubernetes cluster and it: diff --git a/doc/user/clusters/img/kubernetes-agent-ui-list_v13_8.png b/doc/user/clusters/img/kubernetes-agent-ui-list_v13_8.png Binary files differdeleted file mode 100644 index 3f9cc41838a..00000000000 --- a/doc/user/clusters/img/kubernetes-agent-ui-list_v13_8.png +++ /dev/null diff --git a/doc/user/clusters/img/kubernetes-agent-ui-list_v14_5.png b/doc/user/clusters/img/kubernetes-agent-ui-list_v14_5.png Binary files differnew file mode 100644 index 00000000000..3463eeb5d93 --- /dev/null +++ b/doc/user/clusters/img/kubernetes-agent-ui-list_v14_5.png diff --git a/doc/user/clusters/integrations.md b/doc/user/clusters/integrations.md index 70f940c775b..d04cf64d93a 100644 --- a/doc/user/clusters/integrations.md +++ b/doc/user/clusters/integrations.md @@ -4,14 +4,19 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Cluster integrations **(FREE)** +# Cluster integrations (DEPRECATED) **(FREE)** + +> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. GitLab provides several ways to integrate applications to your Kubernetes cluster. To enable cluster integrations, first add a Kubernetes cluster to a GitLab [project](../project/clusters/add_remove_clusters.md) or -[group](../group/clusters/index.md#group-level-kubernetes-clusters) or +[group](../group/clusters/index.md) or [instance](../instance/clusters/index.md). You can install your applications manually as shown in the following sections, or use the @@ -25,10 +30,22 @@ or [Enable Elastic Stack integration for your cluster](#enable-elastic-stack-int depending on which application you are installing. [An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/326565) to automate this step. +Prometheus and Elastic Stack cluster integrations can only be enabled for clusters [connected through cluster certificates](../project/clusters/add_existing_cluster.md). + +To enable Prometheus for your cluster connected through the [GitLab Kubernetes Agent](agent/index.md), you can [integrate it manually](../project/integrations/prometheus.md#manual-configuration-of-prometheus). + +There is no option to enable Elastic Stack for your cluster if it is connected with the GitLab Kubernetes Agent. +Follow this [issue](https://gitlab.com/gitlab-org/gitlab/-/issues/300230) for updates. + ## Prometheus cluster integration > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/55244) in GitLab 13.11. +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. However, you can **still use** Prometheus +for Kubernetes clusters connected to GitLab through the +[GitLab Kubernetes Agent](agent/index.md) by [enabling Prometheus manually](../project/integrations/prometheus.md#manual-configuration-of-prometheus). + You can integrate your Kubernetes cluster with [Prometheus](https://prometheus.io/) for monitoring key metrics of your apps directly from the GitLab UI. diff --git a/doc/user/clusters/management_project.md b/doc/user/clusters/management_project.md index ca6843f6fde..1332310b850 100644 --- a/doc/user/clusters/management_project.md +++ b/doc/user/clusters/management_project.md @@ -4,9 +4,15 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Cluster management project **(FREE)** +# Cluster management project (DEPRECATED) **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32810) in GitLab 12.5 +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32810) in GitLab 12.5. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +The cluster management project was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +To manage cluster applications, use the [GitLab Kubernetes Agent](agent/index.md) +with the [Cluster Management Project Template](management_project_template.md). A project can be designated as the management project for a cluster. A management project can be used to run deployment jobs with @@ -37,8 +43,7 @@ Management projects are restricted to the following: To use a cluster management project to manage your cluster: 1. Create a new project to serve as the cluster management project -for your cluster. We recommend that you -[create this project based on the Cluster Management project template](management_project_template.md#create-a-new-project-based-on-the-cluster-management-template). +for your cluster. 1. [Associate the cluster with the management project](#associate-the-cluster-management-project-with-the-cluster). 1. [Configure your cluster's pipelines](#configuring-your-pipeline). 1. [Set the environment scope](#setting-the-environment-scope). diff --git a/doc/user/clusters/management_project_template.md b/doc/user/clusters/management_project_template.md index 305f66c5ec5..c663246cdd8 100644 --- a/doc/user/clusters/management_project_template.md +++ b/doc/user/clusters/management_project_template.md @@ -4,15 +4,17 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Cluster Management project template **(FREE)** +# Manage cluster applications **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25318) in GitLab 12.10 with Helmfile support via Helm v2. > - Helm v2 support was [dropped](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63577) in GitLab 14.0. Use Helm v3 instead. +> - [Migrated](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/24) to the GitLab Kubernetes Agent in GitLab 14.5. -With a [cluster management project](management_project.md) you can manage -your cluster's deployment and applications through a repository in GitLab. +Use a repository to install, manage, and deploy clusters applications through code. -The Cluster Management project template provides you a baseline to get +## Cluster Management Project Template + +The Cluster Management Project Template provides you a baseline to get started and flexibility to customize your project to your cluster's needs. For instance, you can: @@ -21,49 +23,78 @@ For instance, you can: - Remove the built-in cluster applications you don't need. - Add other cluster applications using the same structure as the ones already available. -The template contains the following [components](#available-components): +The template contains the following [components](#configure-the-available-components): -- A pre-configured GitLab CI/CD file so that you can configure deployment pipelines. +- A pre-configured GitLab CI/CD file so that you can configure CI/CD pipelines using the [CI/CD Tunnel](agent/ci_cd_tunnel.md). - A pre-configured [Helmfile](https://github.com/roboll/helmfile) so that you can manage cluster applications with [Helm v3](https://helm.sh/). - An `applications` directory with a `helmfile.yaml` configured for each application available in the template. -WARNING: -If you used [GitLab Managed Apps](applications.md) to manage your -cluster from GitLab, see how to [migrate from GitLab Managed Apps](migrating_from_gma_to_project_template.md) to the Cluster Management -project. +## Use the Kubernetes Agent with the Cluster Management Project Template + +To use a new project created from the Cluster Management Project Template +with a cluster connected to GitLab through the [GitLab Kubernetes Agent](agent/index.md), +you have two options: + +- [Use one single project](#single-project) to configure the Agent and manage cluster applications. +- [Use separate projects](#separate-projects) - one to configure the Agent and another to manage cluster applications. + +### Single project + +This setup is particularly useful when you haven't connected your cluster +to GitLab through the Agent yet and you want to use the Cluster Management +Project Template to manage cluster applications. + +To use one single project to configure the Agent and to manage cluster applications: + +1. [Create a new project from the Cluster Management Project Template](#create-a-new-project-based-on-the-cluster-management-template). +1. Configure the new project as the [Agent's configuration repository](agent/repository.md) +(where the Agent is registered and its `config.yaml` is stored). +1. From your project's settings, add a [new environment variable](../../ci/variables/index.md#add-a-cicd-variable-to-a-project) `$KUBE_CONTEXT` and set it to `path/to/agent-configuration-project:your-agent-name`. +1. [Configure the components](#configure-the-available-components) inherited from the template. + +### Separate projects + +This setup is particularly useful **when you already have a cluster** connected +to GitLab through the Agent and want to use the Cluster Management +Project Template to manage cluster applications. -## Set up the management project from the Cluster Management project template +To use one project to configure the Agent ("project A") and another project to +manage cluster applications ("project B"), follow the steps below. -To set up your cluster's management project off of the Cluster Management project template: +We assume that you already have a cluster connected through the Agent and +[configured through the Agent's configuration repository](agent/repository.md) +("project A"). -1. [Create a new project based on the Cluster Management template](#create-a-new-project-based-on-the-cluster-management-template). -1. [Associate the cluster management project with your cluster](management_project.md#associate-the-cluster-management-project-with-the-cluster). -1. Use the [available components](#available-components) to manage your cluster. +1. [Create a new project from the Cluster Management Project Template](#create-a-new-project-based-on-the-cluster-management-template). +This new project is "project B". +1. In your "project A", [grant the Agent access to the new project (B) through the CI/CD Tunnel](agent/repository.md#authorize-projects-to-use-an-agent). +1. From the "project's B" settings, add a [new environment variable](../../ci/variables/index.md#add-a-cicd-variable-to-a-project) `$KUBE_CONTEXT` and set it to `path/to/agent-configuration-project:your-agent-name`. +1. In "project B", [configure the components](#configure-the-available-components) inherited from the template. -### Create a new project based on the Cluster Management template +## Create a new project based on the Cluster Management Template To get started, create a new project based on the Cluster Management project template to use as a cluster management project. -You can either create the [new project](../project/working_with_projects.md#create-a-project) -from the template or import the project from the URL. Importing -the project is useful if you are using a GitLab self-managed -instance that may not have the latest version of the template. +You can either create the new project from the template or import the +project from the URL. Importing the project is useful if you are using +a GitLab self-managed instance that may not have the latest version of +the template. -To create the new project: +To [create the new project](../project/working_with_projects.md#create-a-project): - From the template: select the **GitLab Cluster Management** project template. - Importing from the URL: use `https://gitlab.com/gitlab-org/project-templates/cluster-management.git`. -## Available components +## Configure the available components -Use the available components to configure your cluster: +Use the available components to configure your cluster applications: -- [A `.gitlab-ci.yml` file](#the-gitlab-ciyml-file). -- [A main `helmfile.yml` file](#the-main-helmfileyml-file). -- [A directory with built-in applications](#built-in-applications). +- [The `.gitlab-ci.yml` file](#the-gitlab-ciyml-file). +- [The main `helmfile.yml` file](#the-main-helmfileyml-file). +- [The directory with built-in applications](#built-in-applications). ### The `.gitlab-ci.yml` file @@ -107,8 +138,8 @@ The [built-in supported applications](https://gitlab.com/gitlab-org/project-temp - [Sentry](../infrastructure/clusters/manage/management_project_applications/sentry.md) - [Vault](../infrastructure/clusters/manage/management_project_applications/vault.md) -#### How to customize your applications +#### Customize your applications -Each app has an `applications/{app}/values.yaml` file (`applicaton/{app}/values.yaml.gotmpl` in case of GitLab Runner). This is the +Each app has an `applications/{app}/values.yaml` file (`applications/{app}/values.yaml.gotmpl` in case of GitLab Runner). This is the place where you can define default values for your app's Helm chart. Some apps already have defaults pre-defined by GitLab. diff --git a/doc/user/compliance/compliance_report/index.md b/doc/user/compliance/compliance_report/index.md index 63dd06bcfdb..d98a0a145f2 100644 --- a/doc/user/compliance/compliance_report/index.md +++ b/doc/user/compliance/compliance_report/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Compliance report **(ULTIMATE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36524) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.8 as Compliance Dashboard. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/36524) in GitLab 12.8 as Compliance Dashboard. > - [Renamed](https://gitlab.com/gitlab-org/gitlab/-/issues/299360) to compliance report in GitLab 14.2. Compliance report gives you the ability to see a group's merge request activity. It provides a @@ -53,7 +53,7 @@ request: ## Approval status and separation of duties -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217939) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/217939) in GitLab 13.3. We support a separation of duties policy between users who create and approve merge requests. The approval status column can help you identify violations of this policy. @@ -75,9 +75,9 @@ This column has four states: If you see a non-success state, review the criteria for the merge request's project to ensure it complies with the separation of duties. -## Chain of Custody report **(ULTIMATE)** +## Chain of Custody report -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213364) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213364) in GitLab 13.3. The Chain of Custody report allows customers to export a list of merge commits within the group. The data provides a comprehensive view with respect to merge commits. It includes the merge commit SHA, @@ -90,9 +90,9 @@ To download the Chain of Custody report: 1. On the left sidebar, select **Security & Compliance > Compliance report**. 1. Select **List of all merge commits**. -### Commit-specific Chain of Custody Report **(ULTIMATE)** +### Commit-specific Chain of Custody Report -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267629) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/267629) in GitLab 13.6. You can generate a commit-specific Chain of Custody report for a given commit SHA. diff --git a/doc/user/compliance/license_compliance/index.md b/doc/user/compliance/license_compliance/index.md index 5318f4deed1..319c1ca6278 100644 --- a/doc/user/compliance/license_compliance/index.md +++ b/doc/user/compliance/license_compliance/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # License Compliance **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5483) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5483) in GitLab 11.0. If you're using [GitLab CI/CD](../../../ci/index.md), you can use License Compliance to search your project's dependencies for their licenses. You can then decide whether to allow or deny the use of @@ -162,7 +162,7 @@ License Compliance can be configured using CI/CD variables. ### Installing custom dependencies -> Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.4. +> Introduced in GitLab 11.4. The `license_finder` image already embeds many auto-detection scripts, languages, and packages. Nevertheless, it's almost impossible to cover all cases for all projects. @@ -188,6 +188,21 @@ variables: In this example, `my-custom-install-script.sh` is a shell script at the root directory of your project. +### Working with Monorepos + +Depending on your language, you may need to specify the path to the individual +projects of a monorepo using the `LICENSE_FINDER_CLI_OPTS` variable. Passing in +the project paths can significantly speed up builds over using the `--recursive` +license_finder option. + +```yaml +include: + - template: Security/License-Scanning.gitlab-ci.yml + +variables: + LICENSE_FINDER_CLI_OPTS: "--aggregate_paths=relative-path/to/sub-project/one relative-path/to/sub-project/two" +``` + ### Overriding the template WARNING: @@ -262,7 +277,7 @@ License Compliance uses Java 8 by default. You can specify a different Java vers ### Selecting the version of Python -> - [Introduced](https://gitlab.com/gitlab-org/security-products/license-management/-/merge_requests/36) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. +> - [Introduced](https://gitlab.com/gitlab-org/security-products/license-management/-/merge_requests/36) in GitLab 12.0. > - In [GitLab 12.2](https://gitlab.com/gitlab-org/gitlab/-/issues/12032), Python 3.5 became the default. > - In [GitLab 12.7](https://gitlab.com/gitlab-org/security-products/license-management/-/merge_requests/101), Python 3.8 became the default. @@ -695,7 +710,7 @@ Additional configuration may be needed for connecting to private registries for: ### SPDX license list name matching -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/212388) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/212388) in GitLab 13.3. Prior to GitLab 13.3, offline environments required an exact name match for [project policies](#policies). In GitLab 13.3 and later, GitLab matches the name of [project policies](#policies) @@ -705,7 +720,7 @@ instance's administrator can manually update it with a [Rake task](../../../rake ## License list -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13582) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13582) in GitLab 12.7. The License list allows you to see your project's licenses and key details about them. @@ -729,7 +744,7 @@ The licenses are displayed, where: ## Policies -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22465) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/22465) in GitLab 12.9. Policies allow you to specify licenses that are `allowed` or `denied` in a project. If a `denied` license is newly committed it blocks the merge request and instructs the developer to remove it. @@ -752,7 +767,7 @@ Developers of the project can view the policies configured in a project. ## Enabling License Approvals within a project -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13067) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.3. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13067) in GitLab 12.3. Prerequisites: diff --git a/doc/user/discussions/index.md b/doc/user/discussions/index.md index aa4e3ce6f49..4d6cff96169 100644 --- a/doc/user/discussions/index.md +++ b/doc/user/discussions/index.md @@ -10,7 +10,7 @@ type: reference, howto GitLab encourages communication through comments, threads, and [code suggestions](../project/merge_requests/reviews/suggestions.md). -There are two types of comments: +Two types of comments are available: - A standard comment. - A comment in a thread, which can be [resolved](#resolve-a-thread). @@ -119,16 +119,15 @@ Notes are added to the page details. If an issue or merge request is locked and closed, you cannot reopen it. -## Mark a comment as confidential +## Mark a comment as confidential **(FREE SELF)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207473) in GitLab 13.9. -> - [Deployed behind a feature flag](../feature_flags.md), disabled by default. -> - Disabled on GitLab.com. -> - Not recommended for production use. -> - To use in GitLab self-managed instances, ask a GitLab administrator to enable it. **(FREE SELF)** +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207473) in GitLab 13.9 [with a flag](../../administration/feature_flags.md) named `confidential_notes`. Disabled by default. -WARNING: -This feature might not be available to you. Check the **version history** note above for details. +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../../administration/feature_flags.md) named `confidential_notes`. +On GitLab.com, this feature is not available. +You should not use this feature for production environments. You can make a comment confidential, so that it is visible only to project members who have at least the Reporter role. @@ -142,8 +141,6 @@ You can also make an [entire issue confidential](../project/issues/confidential_ ## Show only comments -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/26723) in GitLab 11.5. - For issues and merge requests with many comments, you can filter the page to show comments only. 1. Open a merge request's **Discussion** tab, or epic or issue's **Overview** tab. @@ -171,14 +168,12 @@ You can assign an issue to a user who made a comment. ## Create a thread by replying to a standard comment -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/30299) in GitLab 11.9. - When you reply to a standard comment, you create a thread. Prerequisites: - You must have at least the [Guest role](../permissions.md#project-members-permissions). -- You must be in an issue, merge request, or epic. Commits and snippets threads are not supported. +- You must be in an issue, merge request, or epic. Threads in commits and snippets are not supported. To create a thread by replying to a comment: @@ -186,7 +181,7 @@ To create a thread by replying to a comment:  - The reply area is displayed. + The reply section is displayed. 1. Enter your reply. 1. Select **Comment** or **Add comment now** (depending on where in the UI you are replying). @@ -215,8 +210,7 @@ A threaded comment is created. ## Resolve a thread -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/5022) in GitLab 8.11. -> - Resolving comments individually was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/28750) in GitLab 13.6. +> Resolving comments individually was [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/28750) in GitLab 13.6. In a merge request, you can resolve a thread when you want to finish a conversation. @@ -286,22 +280,3 @@ with a new push. Threads are now resolved if a push makes a diff section outdated. Threads on lines that don't change and top-level resolvable threads are not resolved. - -## Enable or disable confidential comments **(FREE SELF)** - -Confidential comments are under development and not ready for production use. The feature is -deployed behind a feature flag that is **disabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../administration/feature_flags.md) -can enable it. - -To enable it: - -```ruby -Feature.enable(:confidential_notes) -``` - -To disable it: - -```ruby -Feature.disable(:confidential_notes) -``` diff --git a/doc/user/gitlab_com/index.md b/doc/user/gitlab_com/index.md index e0cc79fe0fb..a3aecff6f73 100644 --- a/doc/user/gitlab_com/index.md +++ b/doc/user/gitlab_com/index.md @@ -135,11 +135,13 @@ the related documentation. | Scheduled Pipeline Cron | `*/5 * * * *` | `3-59/10 * * * *` | | [Max jobs in active pipelines](../../administration/instance_limits.md#number-of-jobs-in-active-pipelines) | `500` for Free tier, unlimited otherwise | Unlimited | | [Max CI/CD subscriptions to a project](../../administration/instance_limits.md#number-of-cicd-subscriptions-to-a-project) | `2` | Unlimited | +| [Max number of pipeline triggers in a project](../../administration/instance_limits.md#limit-the-number-of-pipeline-triggers) | `25000` for Free tier, Unlimited for all paid tiers | Unlimited | | [Max pipeline schedules in projects](../../administration/instance_limits.md#number-of-pipeline-schedules) | `10` for Free tier, `50` for all paid tiers | Unlimited | | [Max pipelines per schedule](../../administration/instance_limits.md#limit-the-number-of-pipelines-created-by-a-pipeline-schedule-per-day) | `24` for Free tier, `288` for all paid tiers | Unlimited | | [Scheduled Job Archival](../../user/admin_area/settings/continuous_integration.md#archive-jobs) | 3 months | Never | | Max test cases per [unit test report](../../ci/unit_test_reports.md) | `500_000` | Unlimited | | [Max registered runners](../../administration/instance_limits.md#number-of-registered-runners-per-scope) | Free tier: `50` per-group / `50` per-project <br/> All paid tiers: `1_000` per-group / `1_000` per-project | `1_000` per-group / `1_000` per-project | +| [Limit dotenv variables](../../administration/instance_limits.md#limit-dotenv-variables) | Free tier: `50` / Premium tier: `100` / Ultimate tier: `150` | Unlimited | ## Account and limit settings @@ -158,7 +160,7 @@ If you are near or over the repository size limit, you can either NOTE: `git push` and GitLab project imports are limited to 5 GB per request through Cloudflare. Git LFS and imports other than a file upload are not affected by -this limit. +this limit. Repository limits apply to both public and private projects. ## IP range @@ -198,11 +200,11 @@ The following limits apply for [Webhooks](../project/integrations/webhooks.md): | [Number of webhooks](../../administration/instance_limits.md#number-of-webhooks) | `100` per project, `50` per group | `100` per project, `50` per group | | Maximum payload size | 25 MB | 25 MB | -## Shared Build Cloud runners +## Shared Runner Cloud runners GitLab has shared runners on GitLab.com that you can use to run your CI jobs. -For more information, see [GitLab Build Cloud runners](../../ci/runners/index.md). +For more information, see [GitLab Runner Cloud runners](../../ci/runners/index.md). ## Sidekiq @@ -296,6 +298,7 @@ after the limits change in January, 2021: | **All** traffic (from a given **IP address**) | **600** requests per minute | **2,000** requests per minute | **2,000** requests per minute | | **Issue creation** | | **300** requests per minute | **300** requests per minute | | **Note creation** (on issues and merge requests) | | **300** requests per minute | **60** requests per minute | +| **Advanced, project, and group search** API (for a given **IP address**) | | | **10** requests per minute | More details are available on the rate limits for [protected paths](#protected-paths-throttle) and [raw diff --git a/doc/user/group/clusters/index.md b/doc/user/group/clusters/index.md index 25eff076d8d..9c95b2b21a4 100644 --- a/doc/user/group/clusters/index.md +++ b/doc/user/group/clusters/index.md @@ -5,9 +5,14 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Group-level Kubernetes clusters **(FREE)** +# Group-level Kubernetes clusters (certificate-based) (DEPRECATED) **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34758) in GitLab 11.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34758) in GitLab 11.6. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. To connect clusters to GitLab, +use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). Similar to [project-level](../../project/clusters/index.md) and [instance-level](../../instance/clusters/index.md) Kubernetes clusters, @@ -28,7 +33,7 @@ installation, such as an Ingress controller. ## RBAC compatibility > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/29398) in GitLab 11.4. -> - [Project namespace restriction](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51716) was introduced in GitLab 11.5. +> - Project namespace restriction was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51716) in GitLab 11.5. For each project under a group with a Kubernetes cluster, GitLab creates a restricted service account with [`edit` privileges](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#user-facing-roles) @@ -44,7 +49,7 @@ to the project, provided the cluster is not disabled. ## Multiple Kubernetes clusters -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35094) in GitLab Free 13.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35094) in GitLab 13.2. You can associate more than one Kubernetes cluster to your group, and maintain different clusters for different environments, such as development, staging, and production. diff --git a/doc/user/group/contribution_analytics/index.md b/doc/user/group/contribution_analytics/index.md index b13dd3f63cb..76a8eb77e72 100644 --- a/doc/user/group/contribution_analytics/index.md +++ b/doc/user/group/contribution_analytics/index.md @@ -8,7 +8,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3090) in GitLab 12.2 for subgroups. -With Contribution Analytics you can get an overview of the [contribution actions](../../../api/events.md#action-types) in your +With Contribution Analytics, you can get an overview of the [contribution events](../../index.md#user-contribution-events) in your group. - Analyze your team's contributions over a period of time, and offer a bonus for the top diff --git a/doc/user/group/custom_project_templates.md b/doc/user/group/custom_project_templates.md index a9c56139b4d..9378b3922b5 100644 --- a/doc/user/group/custom_project_templates.md +++ b/doc/user/group/custom_project_templates.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Custom group-level project templates **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6861) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/6861) in GitLab 11.6. [Group owners](../permissions.md#group-members-permissions) can set a subgroup to be the source of project templates that are selectable when a new project is created diff --git a/doc/user/group/devops_adoption/index.md b/doc/user/group/devops_adoption/index.md index e3b858aff7d..36ccfc1031f 100644 --- a/doc/user/group/devops_adoption/index.md +++ b/doc/user/group/devops_adoption/index.md @@ -48,7 +48,7 @@ With DevOps Adoption you can: - Identify specific subgroups that are lagging in their adoption of GitLab, so you can help them along in their DevOps journey. - Find the subgroups that have adopted certain features, and can provide guidance to other subgroups on how to use those features. - + Feature adoption is based on usage in the previous calendar month. Data is updated on the first day of each month. If the monthly update fails, it tries again daily until successful. diff --git a/doc/user/group/epics/index.md b/doc/user/group/epics/index.md index 68df71d1c5d..65e0f31324b 100644 --- a/doc/user/group/epics/index.md +++ b/doc/user/group/epics/index.md @@ -7,8 +7,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Epics **(PREMIUM)** -> - Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.2. -> - Single-level epics [were moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) to [GitLab Premium](https://about.gitlab.com/pricing/) in 12.8. +> - Introduced in GitLab 10.2. +> - Single-level epics were [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) from GitLab Ultimate to GitLab Premium in 12.8. When [issues](../../project/issues/index.md) share a theme across projects and milestones, you can manage them by using epics. @@ -39,7 +39,7 @@ graph TD ## Roadmap in epics **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7327) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7327) in GitLab 11.10. If your epic contains one or more [child epics](manage_epics.md#multi-level-child-epics) that have a start or due date, a visual diff --git a/doc/user/group/epics/manage_epics.md b/doc/user/group/epics/manage_epics.md index 1b36d6f03df..72fa9e1e310 100644 --- a/doc/user/group/epics/manage_epics.md +++ b/doc/user/group/epics/manage_epics.md @@ -12,7 +12,7 @@ to them. ## Create an epic -> - The New Epic form [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211533) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +> - The New Epic form [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211533) in GitLab 13.2. > - In [GitLab 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/229621) and later, the New Epic button on the Epics list opens the New Epic form. > - In [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/45948) and later, you can create a new epic from an empty roadmap. @@ -25,10 +25,11 @@ To create an epic in the group you're in: - In an empty [roadmap](../roadmap/index.md), select **New epic**. 1. Enter a title. -1. Optional. Enter a description. -1. Optional. To make the epic confidential, select the [Confidentiality checkbox](#make-an-epic-confidential). -1. Optional. Choose labels. -1. Optional. Select a start and due date, or [inherit](#start-and-due-date-inheritance) them. +1. Complete the fields. + - Enter a description. + - To [make the epic confidential](#make-an-epic-confidential), select the checkbox under **Confidentiality**. + - Choose labels. + - Select a start and due date, or [inherit](#start-and-due-date-inheritance) them. 1. Select **Create epic**. The newly created epic opens. @@ -69,38 +70,41 @@ After you create an epic, you can edit the following details: To edit an epic's title or description: -1. Select the **Edit title and description** **{pencil}** button. +1. Select **Edit title and description** **{pencil}**. 1. Make your changes. 1. Select **Save changes**. To edit an epic's start date, due date, or labels: -1. Select **Edit** next to each section in the epic sidebar. +1. Next to each section in the right sidebar, select **Edit**. 1. Select the dates or labels for your epic. ## Bulk edit epics -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7250) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7250) in GitLab 12.2. -Users with permission level of [Reporter or higher](../../permissions.md) can manage epics. +Users with at least the [Reporter role](../../permissions.md) can manage epics. When bulk editing epics in a group, you can edit their labels. To update multiple epics at the same time: 1. In a group, go to **Epics > List**. -1. Click **Edit epics**. A sidebar on the right-hand side of your screen appears with editable fields. -1. Check the checkboxes next to each epic you want to edit. +1. Select **Edit epics**. A sidebar on the right appears with editable fields. +1. Select the checkboxes next to each epic you want to edit. 1. Select the appropriate fields and their values from the sidebar. -1. Click **Update all**. +1. Select **Update all**. ## Delete an epic NOTE: -To delete an epic, you need to be an [Owner](../../permissions.md#group-members-permissions) of a group/subgroup. +To delete an epic, you must be an [Owner](../../permissions.md#group-members-permissions) of a group +or subgroup. -When editing the description of an epic, select the **Delete** button to delete the epic. -A modal appears to confirm your action. +To delete the epic: + +1. Select **Edit title and description** **{pencil}**. +1. Select **Delete**. A modal appears to confirm your action. Deleting an epic releases all existing issues from their associated epic in the system. @@ -112,38 +116,56 @@ If you delete an epic, all its child epics and their descendants are deleted as Whenever you decide that there is no longer need for that epic, close the epic by: -- Selecting the **Close epic** button. +- Selecting **Close epic**.  -- Using a [quick action](../../project/quick_actions.md). +- Using the `/close` [quick action](../../project/quick_actions.md). ## Reopen a closed epic You can reopen an epic that was closed by: -- Clicking the **Reopen epic** button. +- Selecting **Reopen epic**.  -- Using a [quick action](../../project/quick_actions.md). +- Using the `/reopen` [quick action](../../project/quick_actions.md). ## Go to an epic from an issue -If an issue belongs to an epic, you can navigate to the containing epic with the -link in the issue sidebar. +If an issue belongs to an epic, you can go to the parent epic with the +link in the right sidebar.  +## View epics list + +In a group, the left sidebar displays the total count of open epics. +This number indicates all epics associated with the group and its subgroups, including epics you +might not have permission to view. + +To view epics in a group: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Epics**. + +### Cached epic count + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299540) in GitLab 13.11 [with a flag](../../../administration/feature_flags.md) named `cached_sidebar_open_epics_count`. Enabled by default. +> - Enabled on self-managed and on GitLab.com in GitLab 14.0. [Feature flag `cached_sidebar_open_epics_count`](https://gitlab.com/gitlab-org/gitlab/-/issues/327320) removed. + +The total count of open epics displayed in the sidebar is cached if higher +than 1000. The cached value is rounded to thousands or millions and updated every 24 hours. + ## Search for an epic from epics list page -> - Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) to the [Premium](https://about.gitlab.com/pricing/) tier in GitLab 12.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) from GitLab Ultimate to GitLab Premium in 12.8. > - Searching by the user's reaction emoji [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/325630) in GitLab 13.11. > - Sorting by epic titles [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/331625) in GitLab 14.1. -You can search for an epic from the list of epics using filtered search bar (similar to -that of issues and merge requests) based on following parameters: +You can search for an epic from the list of epics using filtered search bar based on following +parameters: - Title or description - Author name / username @@ -152,10 +174,13 @@ that of issues and merge requests) based on following parameters:  -To search, go to the list of epics and select the field **Search or filter results**. -It displays a dropdown menu, from which you can add an author. You can also enter plain -text to search by epic title or description. When done, press <kbd>Enter</kbd> on your -keyboard to filter the list. +To search: + +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Epics**. +1. Select the field **Search or filter results**. +1. From the dropdown menu, select the scope or enter plain text to search by epic title or description. +1. Press <kbd>Enter</kbd> on your keyboard. The list is filtered. You can also sort epics list by: @@ -173,22 +198,22 @@ The sort option and order is saved and used wherever you browse epics, including ## Change activity sort order -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214364) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/214364) in GitLab 13.2. You can reverse the default order and interact with the activity feed sorted by most recent items at the top. Your preference is saved via local storage and automatically applied to every epic and issue you view. -To change the activity sort order, click the **Oldest first** dropdown menu and select either oldest +To change the activity sort order, select the **Oldest first** dropdown menu and select either oldest or newest items to be shown first.  ## Make an epic confidential -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213068) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.0 behind a feature flag, disabled by default. -> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/224513) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.2. -> - You can [use the Confidentiality option in the epic sidebar](https://gitlab.com/gitlab-org/gitlab/-/issues/197340) in GitLab [Premium](https://about.gitlab.com/pricing/) 13.3 and later. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213068) in GitLab 13.0 behind a feature flag, disabled by default. +> - [Became enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/224513) in GitLab 13.2. +> - You can [use the Confidentiality option in the epic sidebar](https://gitlab.com/gitlab-org/gitlab/-/issues/197340) in GitLab 13.3 and later. If you're working on items that contain private information, you can make an epic confidential. @@ -200,8 +225,8 @@ to learn how to create a confidential merge request. To make an epic confidential: -- **When creating an epic:** select the checkbox **Make this epic confidential**. -- **In an existing epic:** in the epic's sidebar, select **Edit** next to **Confidentiality** then +- **When creating an epic:** select the checkbox under **Confidentiality**. +- **In an existing epic:** on the right sidebar, select **Edit** next to **Confidentiality**, and then select **Turn on**. ## Manage issues assigned to an epic @@ -233,12 +258,12 @@ current parent. To add a new issue to an epic: -1. On the epic's page, under **Epics and Issues**, select the **Add** dropdown button. +1. On the epic's page, under **Epics and Issues**, select **Add**. 1. Select **Add an existing issue**. 1. Identify the issue to be added, using either of the following methods: - Paste the link of the issue. - Search for the desired issue by entering part of the issue's title, then selecting the desired - match (introduced in [GitLab 12.5](https://gitlab.com/gitlab-org/gitlab/-/issues/9126)). + match ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9126) in GitLab 12.5). If there are multiple issues to be added, press <kbd>Space</kbd> and repeat this step. 1. Select **Add**. @@ -252,7 +277,7 @@ while dividing work into smaller parts. To create an issue from an epic: -1. On the epic's page, under **Epics and Issues**, select the **Add** dropdown button. +1. On the epic's page, under **Epics and Issues**, select **Add**. 1. Select **Add a new issue**. 1. Under **Title**, enter the title for the new issue. 1. From the **Project** dropdown, select the project in which the issue should be created. @@ -265,7 +290,7 @@ After you remove an issue from an epic, the issue is no longer associated with t To remove an issue from an epic: -1. Select the **Remove** (**{close}**) button next to the issue you want to remove. +1. Next to the issue you want to remove, select **Remove** (**{close}**). The **Remove issue** warning appears. 1. Select **Remove**. @@ -285,7 +310,7 @@ To reorder issues assigned to an epic: ### Move issues between epics **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33039) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33039) in GitLab 13.0. New issues appear at the top of the list in the **Epics and Issues** tab. You can move issues from one epic to another. @@ -297,8 +322,8 @@ To move an issue to another epic: ### Promote an issue to an epic -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3777) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.6. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) to [GitLab Premium](https://about.gitlab.com/pricing/) in 12.8. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/3777) in GitLab 11.6. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/37081) from GitLab Ultimate to GitLab Premium in 12.8. If you have the necessary [permissions](../../permissions.md) to close an issue and create an epic in the immediate parent group, you can promote an issue to an epic with the `/promote` @@ -318,7 +343,7 @@ The following issue metadata is copied to the epic: - Upvotes/downvotes. - Participants. - Group labels that the issue already has. -- Parent epic. **(ULTIMATE)** +- Parent epic. ### Use an epic template for repeating issues @@ -331,8 +356,6 @@ For more on epic templates, see [Epic Templates - Repeatable sets of issues](htt ## Multi-level child epics **(ULTIMATE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8333) in GitLab Ultimate 11.7. - You can add any epic that belongs to a group or subgroup of the parent epic's group. New child epics appear at the top of the list of epics in the **Epics and Issues** tab. @@ -344,19 +367,19 @@ Epics can contain multiple nested child epics, up to a total of seven levels dee To add a child epic to an epic: -1. Select the **Add** dropdown button. +1. Select **Add**. 1. Select **Add a new epic**. 1. Identify the epic to be added, using either of the following methods: - Paste the link of the epic. - Search for the desired issue by entering part of the epic's title, then selecting the desired - match (introduced in [GitLab 12.5](https://gitlab.com/gitlab-org/gitlab/-/issues/9126)). + match ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9126) in GitLab 12.5). If there are multiple epics to be added, press <kbd>Space</kbd> and repeat this step. 1. Select **Add**. ### Move child epics between epics -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33039) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 13.0. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/33039) in GitLab 13.0. New child epics appear at the top of the list in the **Epics and Issues** tab. You can move child epics from one epic to another. @@ -384,13 +407,6 @@ To reorder child epics assigned to an epic: To remove a child epic from a parent epic: -1. Select the <kbd>x</kbd> button in the parent epic's list of epics. -1. Select **Remove** in the **Remove epic** warning message. - -## Cached epic count - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299540) in GitLab 13.11. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/327320) in GitLab 14.0. - -In a group, the sidebar displays the total count of open epics and this value is cached if higher -than 1000. The cached value is rounded to thousands (or millions) and updated every 24 hours. +1. Select **Remove** (**{close}**) in the parent epic's list of epics. + The **Remove epic** warning appears. +1. Select **Remove**. diff --git a/doc/user/group/import/index.md b/doc/user/group/import/index.md index 1f5de36303e..70406cfe8e8 100644 --- a/doc/user/group/import/index.md +++ b/doc/user/group/import/index.md @@ -7,9 +7,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Migrate groups from another instance of GitLab **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249160) in GitLab 13.7. -> - [Deployed behind a feature flag](../../feature_flags.md), disabled by default. -> - Enabled on GitLab.com. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/249160) in GitLab 13.7 [with a flag](../../feature_flags.md) named `bulk_import`. Disabled by default. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/338985) in GitLab 14.3. NOTE: The importer migrates **only** the group data listed on this page. To leave feedback on this @@ -19,23 +18,23 @@ Using GitLab Group Migration, you can migrate existing top-level groups from Git The following resources are migrated to the target instance: -- Groups ([Introduced in 13.7](https://gitlab.com/groups/gitlab-org/-/epics/4374)) +- Groups ([Introduced](https://gitlab.com/groups/gitlab-org/-/epics/4374) in 13.7) - description - attributes - subgroups - - avatar ([Introduced in 14.0](https://gitlab.com/gitlab-org/gitlab/-/issues/322904)) -- Group Labels ([Introduced in 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/292429)) + - avatar ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/322904) in 14.0) +- Group Labels ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292429) in 13.9) - title - description - color - - created_at ([Introduced in 13.10](https://gitlab.com/gitlab-org/gitlab/-/issues/300007)) - - updated_at ([Introduced in 13.10](https://gitlab.com/gitlab-org/gitlab/-/issues/300007)) -- Members ([Introduced in 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/299415)) + - created_at ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300007) in 13.10) + - updated_at ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/300007) in 13.10) +- Members ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/299415) in 13.9) Group members are associated with the imported group if: - The user already exists in the target GitLab instance and - The user has a public email in the source GitLab instance that matches a confirmed email in the target GitLab instance -- Epics ([Introduced in 13.7](https://gitlab.com/gitlab-org/gitlab/-/issues/250281)) +- Epics ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/250281) in 13.7) - title - description - state (open / closed) @@ -43,12 +42,12 @@ The following resources are migrated to the target instance: - due date - epic order on boards - confidentiality - - labels ([Introduced in 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/297460)) - - author ([Introduced in 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/298745)) - - parent epic ([Introduced in 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/297459)) - - emoji award ([Introduced in 13.9](https://gitlab.com/gitlab-org/gitlab/-/issues/297466)) - - events ([Introduced in 13.10](https://gitlab.com/gitlab-org/gitlab/-/issues/297465)) -- Milestones ([Introduced in 13.10](https://gitlab.com/gitlab-org/gitlab/-/issues/292427)) + - labels ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/297460) in 13.9) + - author ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/298745) in 13.9) + - parent epic ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/297459) in 13.9) + - emoji award ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/297466) in 13.9) + - events ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/297465) in 13.10) +- Milestones ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292427) in 13.10) - title - description - state (active / closed) @@ -56,8 +55,8 @@ The following resources are migrated to the target instance: - due date - created at - updated at - - iid ([Introduced in 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/326157)) -- Iterations ([Introduced in 13.10](https://gitlab.com/gitlab-org/gitlab/-/issues/292428)) + - iid ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326157) in 13.11) +- Iterations ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292428) in 13.10) - iid - title - description @@ -66,7 +65,7 @@ The following resources are migrated to the target instance: - due date - created at - updated at -- Badges ([Introduced in 13.11](https://gitlab.com/gitlab-org/gitlab/-/issues/292431)) +- Badges ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292431) in 13.11) - name - link URL - image URL @@ -77,19 +76,20 @@ Any other items are **not** migrated. ## Enable or disable GitLab Group Migration -GitLab Migration is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) can enable it. +GitLab Migration is deployed behind the `bulk_import` feature flag, which is **enabled by default**. +[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) +can disable it. -To enable it: +To disable it: ```ruby -Feature.enable(:bulk_import) +Feature.disable(:bulk_import) ``` -To disable it: +To enable it: ```ruby -Feature.disable(:bulk_import) +Feature.enable(:bulk_import) ``` ## Import your groups into GitLab @@ -130,3 +130,8 @@ Migration importer page. The remote groups you have the Owner role for are liste 1. After a group has been imported, select its GitLab path to open its GitLab URL.  + +## Automate group and project import **(PREMIUM)** + +For information on automating user, group, and project import API calls, see +[Automate group and project import](../../project/import/index.md#automate-group-and-project-import). diff --git a/doc/user/group/index.md b/doc/user/group/index.md index 15d2da50012..f0e08301a1b 100644 --- a/doc/user/group/index.md +++ b/doc/user/group/index.md @@ -95,10 +95,8 @@ For details about groups, watch [GitLab Namespaces (users, groups and subgroups) You can give a user access to all projects in a group. -1. On the top bar, select **Menu > Groups**. -1. Select **Your Groups**. -1. Find your group and select it. -1. From the left sidebar, select **Group information > Members**. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Group information > Members**. 1. Fill in the fields. - The role applies to all projects in the group. [Learn more about permissions](../permissions.md). - On the **Access expiration date**, the user can no longer access projects in the group. @@ -107,9 +105,7 @@ You can give a user access to all projects in a group. As a user, you can request to be a member of a group, if an administrator allows it. -1. On the top bar, select **Menu > Groups**. -1. Select **Your Groups**. -1. Find the group and select it. +1. On the top bar, select **Menu > Groups** and find your group. 1. Under the group name, select **Request Access**. As many as ten of the most-recently-active group owners receive an email with your request. @@ -219,10 +215,13 @@ By default, every group inherits the branch protection set at the global level. To change this setting for a specific group: -1. Go to the group's **Settings > General** page. +1. On the top bar, select **Menu > Groups**. +1. Select **Your Groups**. +1. Find the group and select it. +1. From the left menu, select **Settings > General**. 1. Expand the **Permissions, LFS, 2FA** section. 1. Select the desired option in the **Default branch protection** dropdown list. -1. Click **Save changes**. +1. Select **Save changes**. To change this setting globally, see [Default branch protection](../admin_area/settings/visibility_and_access_controls.md#protect-default-branches). @@ -240,24 +239,26 @@ There are two different ways to add a new project to a group: ### Specify who can add projects to a group -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2534) in GitLab Premium 10.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/25975) to GitLab Free in 11.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/2534) in GitLab 10.5. +> - [Moved](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/25975) from GitLab Premium to GitLab Free in 11.10. By default, [Developers and Maintainers](../permissions.md#group-members-permissions) can create projects under a group. To change this setting for a specific group: -1. Go to the group's **Settings > General** page. +1. On the top bar, select **Menu > Groups**. +1. Select **Your Groups**. +1. Find the group and select it. +1. From the left menu, select **Settings > General**. 1. Expand the **Permissions, LFS, 2FA** section. 1. Select the desired option in the **Allowed to create projects** dropdown list. -1. Click **Save changes**. +1. Select **Save changes**. To change this setting globally, see [Default project creation protection](../admin_area/settings/visibility_and_access_controls.md#define-which-roles-can-create-projects). ## Group activity analytics **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207164) in GitLab 12.10 as -a [beta feature](https://about.gitlab.com/handbook/product/#beta). +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/207164) in GitLab 12.10 as a [beta feature](https://about.gitlab.com/handbook/product/#beta). For a group, you can view how many merge requests, issues, and members were created in the last 90 days. @@ -265,6 +266,8 @@ These Group Activity Analytics can be enabled with the `group_activity_analytics  +Changes to [group wikis](../project/wiki/group.md) do not appear in group activity analytics. + ### View group activity You can view the most recent actions taken in a group, either in your browser or in an RSS feed: @@ -307,7 +310,7 @@ All the members of the `Engineering` group are added to the `Frontend` group. > - Enabled on GitLab.com. > - Recommended for production use. > - Replaces the existing form with buttons to open a modal window. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](../project/members/index.md#enable-or-disable-modal-window). **(FREE SELF)** +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](../project/members/index.md#enable-or-disable-modal-window). WARNING: This feature might not be available to you. Check the **version history** note above for details. @@ -327,7 +330,7 @@ Group syncing allows LDAP groups to be mapped to GitLab groups. This provides mo Group links can be created by using either a CN or a filter. To create these group links, go to the group's **Settings > LDAP Synchronization** page. After configuring the link, it may take more than an hour for the users to sync with the GitLab group. -For more information on the administration of LDAP and group sync, refer to the [main LDAP documentation](../../administration/auth/ldap/index.md#group-sync). +For more information on the administration of LDAP and group sync, refer to the [main LDAP documentation](../../administration/auth/ldap/ldap_synchronization.md#group-sync). NOTE: When you add LDAP synchronization, if an LDAP user is a group member and they are not part of the LDAP group, they are removed from the group. @@ -410,7 +413,7 @@ because the project cannot be moved. ## Use a custom name for the initial branch -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43290) in GitLab 13.6. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/43290) in GitLab 13.6. When you create a new project in GitLab, a default branch is created with the first push. The group owner can @@ -430,7 +433,7 @@ This action removes the group. It also adds a background job to delete all proje Specifically: -- In [GitLab 12.8 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/33257), on [Premium](https://about.gitlab.com/pricing/premium/) or higher tiers, this action adds a background job to mark a group for deletion. By default, the job schedules the deletion 7 days in the future. You can modify this waiting period through the [instance settings](../admin_area/settings/visibility_and_access_controls.md#default-deletion-delay). +- In [GitLab 12.8 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/33257), on [GitLab Premium](https://about.gitlab.com/pricing/premium/) or higher tiers, this action adds a background job to mark a group for deletion. By default, the job schedules the deletion 7 days in the future. You can modify this waiting period through the [instance settings](../admin_area/settings/visibility_and_access_controls.md#default-deletion-delay). - In [GitLab 13.6 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/39504), if the user who sets up the deletion is removed from the group before the deletion happens, the job is cancelled, and the group is no longer scheduled for deletion. @@ -502,6 +505,9 @@ To prevent a project from being shared with other groups: 1. Select **Prevent sharing a project within `<group_name>` with other groups**. 1. Select **Save changes**. +This setting applies to all subgroups unless overridden by a group owner. Groups already +added to a project lose access when the setting is enabled. + ## Prevent members from being added to a group **(PREMIUM)** As a group owner, you can prevent any new project membership for all @@ -522,10 +528,8 @@ API requests to add a new user to a project are not possible. ## Export members as CSV **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/287940) in GitLab 14.2. - -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature per group, ask an administrator to [disable the :ff_group_membership_export flag](../../administration/feature_flags.md). On GitLab.com, this feature is available. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/287940) in GitLab 14.2. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/336520) in GitLab 14.5. You can export a list of members in a group as a CSV. @@ -535,8 +539,8 @@ You can export a list of members in a group as a CSV. ## Restrict group access by IP address **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1985) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/215410) to [GitLab Premium](https://about.gitlab.com/pricing/) in 13.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1985) in GitLab 12.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/215410) from GitLab Ultimate to GitLab Premium in 13.1. To ensure only people from your organization can access particular resources, you can restrict access to groups by IP address. This group-level setting @@ -571,7 +575,7 @@ To restrict group access by IP address: ## Restrict group access by domain **(PREMIUM)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7297) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7297) in GitLab 12.2. > - Support for specifying multiple email domains [added](https://gitlab.com/gitlab-org/gitlab/-/issues/33143) in GitLab 13.1. > - Support for restricting access to projects within the group [added](https://gitlab.com/gitlab-org/gitlab/-/issues/14004) in GitLab 14.1.2. @@ -614,7 +618,7 @@ To learn how to create templates for issues and merge requests, see [Description templates](../project/description_templates.md). Define project templates at a group level by setting a group as the template source. -[Learn more about group-level project templates](custom_project_templates.md). **(PREMIUM)** +[Learn more about group-level project templates](custom_project_templates.md). ### Enable group file template **(PREMIUM)** @@ -693,7 +697,7 @@ In GitLab 13.11 and above the group setting for delayed project removal is inher > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216987) in GitLab 13.3. By default, projects in a group can be forked. -Optionally, on [Premium](https://about.gitlab.com/pricing/) or higher tiers, +Optionally, on [GitLab Premium](https://about.gitlab.com/pricing/) or higher tiers, you can prevent the projects in a group from being forked outside of the current top-level group. Previously, this setting was available only for groups enforcing a @@ -732,19 +736,17 @@ The group's new subgroups have push rules set for them based on either: ## Group approval rules **(PREMIUM)** -> Introduced in GitLab 13.9. [Deployed behind the `group_merge_request_approval_settings_feature_flag` flag](../../administration/feature_flags.md), disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285458) in GitLab 13.9. [Deployed behind the `group_merge_request_approval_settings_feature_flag` flag](../../administration/feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/285410) in GitLab 14.5. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, -per group, ask an administrator to [enable the `group_merge_request_approval_settings_feature_flag` flag](../../administration/feature_flags.md). -The feature is not ready for production use. +On self-managed GitLab, by default this feature is available. To hide the feature per group, ask an administrator to [disable the feature flag](../../administration/feature_flags.md) named `group_merge_request_approval_settings_feature_flag`. On GitLab.com, this feature is available. -Group approval rules are an in-development feature that provides an interface for managing -[project merge request approval rules](../project/merge_requests/approvals/index.md) at the -top-level group level. When rules are configured [at the instance level](../admin_area/merge_requests_approvals.md), -you can't edit locked rules. +Group approval rules manage [project merge request approval rules](../project/merge_requests/approvals/index.md) +at the top-level group level. These rules [cascade to all projects](../project/merge_requests/approvals/settings.md#settings-cascading) +that belong to the group. -To view the merge request approval rules UI for a group: +To view the merge request approval rules for a group: 1. Go to the top-level group's **Settings > General** page. 1. Expand the **Merge request approvals** section. @@ -754,20 +756,20 @@ To view the merge request approval rules UI for a group: ## Related topics - [Group wikis](../project/wiki/index.md) -- [Maximum artifacts size](../admin_area/settings/continuous_integration.md#maximum-artifacts-size). **(FREE SELF)** -- [Repositories analytics](repositories_analytics/index.md): View overall activity of all projects with code coverage. **(PREMIUM)** +- [Maximum artifacts size](../admin_area/settings/continuous_integration.md#maximum-artifacts-size). +- [Repositories analytics](repositories_analytics/index.md): View overall activity of all projects with code coverage. - [Contribution analytics](contribution_analytics/index.md): View the contributions (pushes, merge requests, - and issues) of group members. **(PREMIUM)** -- [Issue analytics](issues_analytics/index.md): View a bar chart of your group's number of issues per month. **(PREMIUM)** + and issues) of group members. +- [Issue analytics](issues_analytics/index.md): View a bar chart of your group's number of issues per month. - Use GitLab as a [dependency proxy](../packages/dependency_proxy/index.md) for upstream Docker images. -- [Epics](epics/index.md): Track groups of issues that share a theme. **(ULTIMATE)** +- [Epics](epics/index.md): Track groups of issues that share a theme. - [Security Dashboard](../application_security/security_dashboard/index.md): View the vulnerabilities of all - the projects in a group and its subgroups. **(ULTIMATE)** + the projects in a group and its subgroups. - [Insights](insights/index.md): Configure insights like triage hygiene, issues created/closed per a given period, and - average time for merge requests to be merged. **(ULTIMATE)** + average time for merge requests to be merged. - [Webhooks](../project/integrations/webhooks.md). - [Kubernetes cluster integration](clusters/index.md). -- [Audit Events](../../administration/audit_events.md#group-events). **(PREMIUM)** +- [Audit Events](../../administration/audit_events.md#group-events). - [Pipelines quota](../admin_area/settings/continuous_integration.md): Keep track of the pipeline quota for the group. - [Integrations](../admin_area/settings/project_integration_management.md). - [Transfer a project into a group](../project/settings/index.md#transferring-an-existing-project-into-another-namespace). @@ -775,7 +777,7 @@ To view the merge request approval rules UI for a group: - [Lock the sharing with group feature](#prevent-a-project-from-being-shared-with-groups). - [Enforce two-factor authentication (2FA)](../../security/two_factor_authentication.md#enforce-2fa-for-all-users-in-a-group): Enforce 2FA for all group members. -- Namespaces [API](../../api/namespaces.md) and [Rake tasks](../../raketasks/features.md).. +- Namespaces [API](../../api/namespaces.md) and [Rake tasks](../../raketasks/features.md). ## Troubleshooting diff --git a/doc/user/group/insights/index.md b/doc/user/group/insights/index.md index 9f841691eb8..b3fdeb0ab41 100644 --- a/doc/user/group/insights/index.md +++ b/doc/user/group/insights/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Insights **(ULTIMATE)** -> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 12.0. +> [Introduced](https://gitlab.com/groups/gitlab-org/-/epics/725) in GitLab 12.0. Configure the Insights that matter for your groups. Explore data such as triage hygiene, issues created or closed for a given period, average time for merge diff --git a/doc/user/group/issues_analytics/index.md b/doc/user/group/issues_analytics/index.md index 3d28ef5306d..ac5df6b052f 100644 --- a/doc/user/group/issues_analytics/index.md +++ b/doc/user/group/issues_analytics/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Issue Analytics **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7478) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.5. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7478) in GitLab 11.5. Issue Analytics is a bar graph which illustrates the number of issues created each month. The default time span is 13 months, which includes the current month, and the 12 months diff --git a/doc/user/group/iterations/index.md b/doc/user/group/iterations/index.md index 70fa3ba639d..7a1cbe86d58 100644 --- a/doc/user/group/iterations/index.md +++ b/doc/user/group/iterations/index.md @@ -13,7 +13,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w > - Enabled on GitLab.com. > - Can be enabled or disabled per-group. > - Recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-iterations). **(PREMIUM ONLY)** +> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-iterations). > - Moved to GitLab Premium in 13.9. Iterations are a way to track issues over a period of time. This allows teams @@ -38,7 +38,7 @@ In GitLab, iterations are similar to milestones, with a few differences: > - Deployed behind a [feature flag](../../feature_flags.md), disabled by default. > - Disabled on GitLab.com. > - Not recommended for production use. -> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-iteration-cadences). **(PREMIUM SELF)** +> - To use in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-or-disable-iteration-cadences). This in-development feature might not be available for your use. There can be [risks when enabling features still in development](../../../administration/feature_flags.md#risks-when-enabling-features-still-in-development). @@ -144,7 +144,7 @@ To view an iteration report, go to the iterations list page and select an iterat ### Iteration burndown and burnup charts -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222750) in GitLab 13.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/222750) in GitLab 13.6. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/269972) in GitLab 13.7. The iteration report includes [burndown and burnup charts](../../project/milestones/burndown_and_burnup_charts.md), diff --git a/doc/user/group/roadmap/index.md b/doc/user/group/roadmap/index.md index 656c40d1915..206f3172170 100644 --- a/doc/user/group/roadmap/index.md +++ b/doc/user/group/roadmap/index.md @@ -7,11 +7,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Roadmap **(PREMIUM)** -> - Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.5. -> - In [GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/198062), Roadmaps were moved to the Premium tier. +> - Introduced in GitLab 10.5. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/198062) from GitLab Ultimate to GitLab Premium in 12.9. > - In [GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/5164) and later, the epic bars show epics' title, progress, and completed weight percentage. > - Milestones appear in roadmaps in [GitLab 12.10](https://gitlab.com/gitlab-org/gitlab/-/issues/6802), and later. -> - Feature flag for milestones visible in roadmaps removed in [GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29641). +> - Feature flag for milestones visible in roadmaps [removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/29641) in GitLab 13.0. > - In [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/214375) and later, the Roadmap also shows milestones in projects in a group. > - In [GitLab 13.2](https://gitlab.com/gitlab-org/gitlab/-/issues/212494) and later, milestone bars can be collapsed and expanded. @@ -36,13 +36,10 @@ toggle the list of the milestone bars. ## Sort and filter the Roadmap -> - Filtering roadmaps by milestone [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218621) in GitLab 13.7. -> - Filtering roadmaps by milestone is [deployed behind a feature flag](../../feature_flags.md), enabled by default. -> - Filtering roadmaps by milestone is enabled on GitLab.com. -> - Filtering roadmaps by milestone is recommended for production use. -> - For GitLab self-managed instances, GitLab administrators can opt to [disable it](#enable-or-disable-filtering-roadmaps-by-milestone). **(PREMIUM SELF)** +> - Filtering by milestone [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218621) in GitLab 13.7 [with a flag](../../../administration/feature_flags.md) named `roadmap_daterange_filter`. Enabled by default. > - Filtering by epic confidentiality [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218624) in GitLab 13.9. > - Filtering by epic [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218623) in GitLab 13.11. +> - Filtering by milestone [feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/323917) in GitLab 14.5. WARNING: Filtering roadmaps by milestone might not be available to you. Check the **version history** note above for details. @@ -56,8 +53,6 @@ A dropdown menu lets you show only open or closed epics. By default, all epics a You can sort epics in the Roadmap view by: -- Created date -- Last updated - Start date - Due date @@ -77,40 +72,16 @@ You can also filter epics in the Roadmap view by the epics': Roadmaps can also be [visualized inside an epic](../epics/index.md#roadmap-in-epics). -### Enable or disable filtering roadmaps by milestone **(PREMIUM SELF)** - -Filtering roadmaps by milestone is under development but ready for production use. -It is deployed behind a feature flag that is **enabled by default**. -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can opt to disable it. - -To enable it: - -```ruby -Feature.enable(:async_filtering) -``` - -To disable it: - -```ruby -Feature.disable(:async_filtering) -``` - ## Timeline duration -> - Introduced in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. -> - In [GitLab 12.9](https://gitlab.com/gitlab-org/gitlab/-/issues/198062), Timelines were moved to the Premium tier. +> - Introduced in GitLab 11.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/198062) from GitLab Ultimate to GitLab Premium in 12.9. ### Date range presets > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/204994) in GitLab 14.3. [Deployed behind the `roadmap_daterange_filter` flag](../../../administration/feature_flags.md), disabled by default. -> - [Enabled on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/323917) in GitLab 14.3. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per group, -ask an administrator to [enable the `roadmap_daterange_filter` flag](../../../administration/feature_flags.md). -On GitLab.com, this feature is available. -The feature is ready for production use. +> - [Enabled on GitLab.com and self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/323917) in GitLab 14.3. +> - [Feature flag `roadmap_daterange_filter` removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/72419) in GitLab 14.5. Roadmap provides three date range options, each with predetermined timeline duration: diff --git a/doc/user/group/saml_sso/index.md b/doc/user/group/saml_sso/index.md index 1c894550a14..402007b85b2 100644 --- a/doc/user/group/saml_sso/index.md +++ b/doc/user/group/saml_sso/index.md @@ -118,8 +118,9 @@ SSO has the following effects when enabled: - For groups, users can't share a project in the group outside the top-level group, even if the project is forked. -- For a Git activity, users must be signed-in through SSO before they can push to or - pull from a GitLab repository. +- For Git activity over SSH and HTTPS, users must have at least one active session signed-in through SSO before they can push to or + pull from a GitLab repository. +- Credentials that are not tied to regular users (for example, access tokens and deploy keys) do not have the SSO check enforced. - Users must be signed-in through SSO before they can pull images using the [Dependency Proxy](../../packages/dependency_proxy/index.md). <!-- Add bullet for API activity when https://gitlab.com/gitlab-org/gitlab/-/issues/9152 is complete --> @@ -242,11 +243,12 @@ On subsequent visits, you should be able to go [sign in to GitLab.com with SAML] ### Configure user settings from SAML response -[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/263661) in GitLab 13.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/263661) in GitLab 13.7. GitLab allows setting certain user attributes based on values from the SAML response. -This affects users created on first sign-in via Group SAML. Existing users' -attributes are not affected regardless of the values sent in the SAML response. +Existing users will have these attributes updated if the user was originally +provisioned by the group. Users are provisioned by the group when the account was +created via [SCIM](scim_setup.md) or by first sign-in with SAML SSO for GitLab.com groups. #### Supported user attributes @@ -341,9 +343,8 @@ Ensure your SAML identity provider sends an attribute statement named `Groups` o ``` NOTE: +The value for `Groups` or `groups` in the SAML response can be either the group name or the group ID. To inspect the SAML response, you can use one of these [SAML debugging tools](#saml-debugging-tools). -Also note that the value for `Groups` or `groups` in the SAML response can be either the group name or -the group ID depending what the IdP sends to GitLab. When SAML SSO is enabled for the top-level group, `Maintainer` and `Owner` level users see a new menu item in group **Settings > SAML Group Links**. You can configure one or more **SAML Group Links** to map @@ -516,6 +517,13 @@ Here are possible causes and solutions: | ---------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------ | | When a user account with the email address already exists in GitLab, but the user does not have the SAML identity tied to their account. | The user needs to [link their account](#user-access-and-management). | +User accounts are created in one of the following ways: + +- User registration +- Sign in through OAuth +- Sign in through SAML +- SCIM provisioning + ### Message: "SAML authentication failed: Extern UID has already been taken, User has already been taken" Getting both of these errors at the same time suggests the NameID capitalization provided by the identity provider didn't exactly match the previous value for that user. @@ -531,6 +539,16 @@ Alternatively, the SAML response may be missing the `InResponseTo` attribute in The identity provider administrator should ensure that the login is initiated by the service provider and not the identity provider. +### Message: "Login to a GitLab account to link with your SAML identity" + +A user can see this message when they are trying to [manually link SAML to their existing GitLab.com account](#linking-saml-to-your-existing-gitlabcom-account). + +To resolve this problem, the user should check they are using the correct GitLab password to log in. They first need to +[reset their password](https://gitlab.com/users/password/new) if both: + +- The account was provisioned by SCIM. +- This is the first time the user has logged in the username and password. + ### Stuck in a login "loop" Ensure that the **GitLab single sign-on URL** has been configured as "Login URL" (or similarly named field) in the identity provider's SAML app. diff --git a/doc/user/group/saml_sso/scim_setup.md b/doc/user/group/saml_sso/scim_setup.md index 5e90501d487..dd4558b4a3e 100644 --- a/doc/user/group/saml_sso/scim_setup.md +++ b/doc/user/group/saml_sso/scim_setup.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # SCIM provisioning using SAML SSO for GitLab.com groups **(PREMIUM SAAS)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/9388) in GitLab Premium 11.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/9388) in GitLab 11.10. System for Cross-domain Identity Management (SCIM), is an open standard that enables the automation of user provisioning. When SCIM is provisioned for a GitLab group, membership of @@ -35,9 +35,10 @@ The following identity providers are supported: Once [Group Single Sign-On](index.md) has been configured, we can: -1. Navigate to the group and click **Administration > SAML SSO**. -1. Click on the **Generate a SCIM token** button. -1. Save the token and URL so they can be used in the next step. +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Settings > SAML SSO**. +1. Select **Generate a SCIM token**. +1. Save the token and URL for use in the next step.  @@ -50,14 +51,14 @@ Once [Group Single Sign-On](index.md) has been configured, we can: The SAML application that was created during [Single sign-on](index.md) setup for [Azure](https://docs.microsoft.com/en-us/azure/active-directory/manage-apps/view-applications-portal) now needs to be set up for SCIM. -1. Set up automatic provisioning and administrative credentials by following the +1. Enable automatic provisioning and administrative credentials by following the [Azure's SCIM setup documentation](https://docs.microsoft.com/en-us/azure/active-directory/app-provisioning/use-scim-to-provision-users-and-groups#provisioning-users-and-groups-to-applications-that-support-scim). During this configuration, note the following: -- The `Tenant URL` and `secret token` are the ones retrieved in the +- The `Tenant URL` and `secret token` are the items retrieved in the [previous step](#gitlab-configuration). -- It is recommended to set a notification email and check the **Send an email notification when a failure occurs** checkbox. +- We recommend setting a notification email and selecting the **Send an email notification when a failure occurs** checkbox. - For mappings, we only leave `Synchronize Azure Active Directory Users to AppName` enabled. `Synchronize Azure Active Directory Groups to AppName` is usually disabled. However, this does not mean Azure AD users cannot be provisioned in groups. Leaving it enabled does not break @@ -113,29 +114,27 @@ Make sure that the Okta setup matches our documentation exactly, especially the configuration. Otherwise, the Okta SCIM app may not work properly. 1. Sign in to Okta. -1. If you see an **Admin** button in the top right, click the button. This will - ensure you are in the Admin area. +1. Ensure you are in the Admin section by selecting the **Admin** button located in the top right. The admin button is not visible from the admin page. NOTE: - If you're using the Developer Console, click **Developer Console** in the top - bar and select **Classic UI**. Otherwise, you may not see the buttons described - in the following steps: + If you're using the Developer Console, select **Developer Console** in the top + bar and then select **Classic UI**. Otherwise, you may not see the buttons described in the following steps: -1. In the **Application** tab, click **Add Application**. -1. Search for **GitLab**, find and click on the 'GitLab' application. -1. On the GitLab application overview page, click **Add**. +1. In the **Application** tab, select **Add Application**. +1. Search for **GitLab**, find and select on the 'GitLab' application. +1. On the GitLab application overview page, select **Add**. 1. Under **Application Visibility** select both checkboxes. Currently the GitLab application does not support SAML authentication so the icon should not be shown to users. -1. Click **Done** to finish adding the application. -1. In the **Provisioning** tab, click **Configure API integration**. +1. Select **Done** to finish adding the application. +1. In the **Provisioning** tab, select **Configure API integration**. 1. Select **Enable API integration**. - For **Base URL** enter the URL obtained from the GitLab SCIM configuration page - For **API Token** enter the SCIM token obtained from the GitLab SCIM configuration page -1. Click 'Test API Credentials' to verify configuration. -1. Click **Save** to apply the settings. -1. After saving the API integration details, new settings tabs appear on the left. Choose **To App**. -1. Click **Edit**. -1. Check the box to **Enable** for both **Create Users** and **Deactivate Users**. -1. Click **Save**. +1. Select 'Test API Credentials' to verify configuration. +1. Select **Save** to apply the settings. +1. After saving the API integration details, new settings tabs appear on the left. Select **To App**. +1. Select **Edit**. +1. Select the **Enable** checkbox for both **Create Users** and **Deactivate Users**. +1. Select **Save**. 1. Assign users in the **Assignments** tab. Assigned users are created and managed in your GitLab group. @@ -147,8 +146,8 @@ application described above. ### OneLogin -OneLogin provides a "GitLab (SaaS)" app in their catalog, which includes a SCIM integration. -As the app is developed by OneLogin, please reach out to OneLogin if you encounter issues. +As the developers of this app, OneLogin provides a "GitLab (SaaS)" app in their catalog, which includes a SCIM integration. +Please reach out to OneLogin if you encounter issues. ## User access and linking setup @@ -177,8 +176,8 @@ As long as [Group SAML](index.md) has been configured, existing GitLab.com users - By following these steps: 1. Sign in to GitLab.com if needed. - 1. Click on the GitLab app in the identity provider's dashboard or visit the **GitLab single sign-on URL**. - 1. Click on the **Authorize** button. + 1. In the identity provider's dashboard select the GitLab app or visit the **GitLab single sign-on URL**. + 1. Select the **Authorize**. We recommend users do this prior to turning on sync, because while synchronization is active, there may be provisioning errors for existing users. diff --git a/doc/user/group/settings/import_export.md b/doc/user/group/settings/import_export.md index 3f9d94f044e..3692a2636ab 100644 --- a/doc/user/group/settings/import_export.md +++ b/doc/user/group/settings/import_export.md @@ -40,7 +40,7 @@ be imported into the desired group structure. - To preserve the member list and their respective permissions on imported groups, review the users in these groups. Make sure these users exist before importing the desired groups. -### Exported Contents +### Exported contents The following items are exported: @@ -51,8 +51,8 @@ The following items are exported: - Subgroups (including all the aforementioned data) - Epics - Events -- [Wikis](../../project/wiki/index.md#group-wikis) **(PREMIUM SELF)** - (Introduced in [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247)) +- [Wikis](../../project/wiki/group.md) + ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247) in GitLab 13.9) The following items are **not** exported: @@ -109,6 +109,11 @@ The Enterprise Edition retains some group data that isn't part of the Community Your newly imported group page appears after the operation completes. +## Automate group and project import **(PREMIUM)** + +For information on automating user, group, and project import API calls, see +[Automate group and project import](../../project/import/index.md#automate-group-and-project-import). + ## Version history ### 14.0+ diff --git a/doc/user/group/subgroups/index.md b/doc/user/group/subgroups/index.md index 49032d0a2ef..acce296da93 100644 --- a/doc/user/group/subgroups/index.md +++ b/doc/user/group/subgroups/index.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: reference, howto, concepts --- -# Subgroups +# Subgroups **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/2772) in GitLab 9.0. diff --git a/doc/user/group/value_stream_analytics/index.md b/doc/user/group/value_stream_analytics/index.md index 5c8393f30ab..a0a13c71d95 100644 --- a/doc/user/group/value_stream_analytics/index.md +++ b/doc/user/group/value_stream_analytics/index.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Value Stream Analytics **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196455) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.9 for groups. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/196455) in GitLab 12.9 for groups. Value Stream Analytics measures the time spent to go from an [idea to production](https://about.gitlab.com/blog/2016/08/05/continuous-integration-delivery-and-deployment-with-gitlab/#from-idea-to-production-with-gitlab) @@ -27,6 +27,7 @@ To view group-level Value Stream Analytics: Value Stream Analytics at the group level includes data for the selected group and its subgroups. +NOTE: [Project-level Value Stream Analytics](../../analytics/value_stream_analytics.md) is also available. ## Default stages @@ -77,16 +78,41 @@ Data is shown for workflow items created during the selected date range. To filt 1. Optionally select a project. 1. Select a date range using the available date pickers. +### Upcoming date filter change + +In the [epics](https://gitlab.com/groups/gitlab-org/-/epics/6046), we plan to alter +the date filter behavior to filter the end event time of the currently selected stage. + +The change makes it possible to get a much better picture about the completed items within the +stage and helps uncover long-running items. + +For example, an issue was created a year ago and the current stage was finished in the current month. +If you were to look at the metrics for the last three months, this issue would not be included in the calculation of +the stage metrics. With the new date filter, this item would be included. + +DISCLAIMER: +This page contains information related to upcoming products, features, and functionality. +It is important to note that the information presented is for informational purposes only. +Please do not rely on this information for purchasing or planning purposes. +As with all projects, the items mentioned on this page are subject to change or delay. +The development, release, and timing of any products, features, or functionality remain at the +sole discretion of GitLab Inc. + ## How metrics are measured -> DORA API-based deployment metrics [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/337256) -> to Premium in GitLab 14.3 for group-level Value Stream Analytics. +> DORA API-based deployment metrics for group-level Value Stream Analytics were +> [moved](https://gitlab.com/gitlab-org/gitlab/-/issues/337256) from GitLab Ultimate to GitLab Premium in 14.3. The "Time" metrics near the top of the page are measured as follows: - **Lead time**: median time from issue created to issue closed. - **Cycle time**: median time from first commit to issue closed. (You can associate a commit with an issue by [crosslinking in the commit message](../../project/issues/crosslinking_issues.md#from-commit-messages).) +- **Lead Time for Changes**: median time between when a merge request is merged and deployed to a +production environment for all merge requests deployed in the given time period. +[Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340150) in GitLab 14.5. + +- **Lead Time for Changes**: median duration between merge request merge and deployment to a production environment for all MRs deployed in the given time period. [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/340150) in GitLab 14.5. The "Recent Activity" metrics near the top of the page are measured as follows: @@ -391,8 +417,8 @@ To delete a custom value stream: ## Days to completion chart > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/21631) in GitLab 12.6. -> - [Chart median line removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235455) in GitLab 13.4. -> - [Totals replaced with averages](https://gitlab.com/gitlab-org/gitlab/-/issues/262070) in GitLab 13.12. +> - Chart median line [removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235455) in GitLab 13.4. +> - Totals [replaced](https://gitlab.com/gitlab-org/gitlab/-/issues/262070) with averages in GitLab 13.12. This chart visually depicts the average number of days it takes for cycles to be completed. @@ -404,7 +430,7 @@ The chart data is limited to the last 500 items. ## Type of work - Tasks by type chart -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32421) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/32421) in GitLab 12.10. This chart shows a cumulative count of issues and merge requests per day. diff --git a/doc/user/index.md b/doc/user/index.md index c9f20af9668..a3b7cfc4b3c 100644 --- a/doc/user/index.md +++ b/doc/user/index.md @@ -85,6 +85,7 @@ There are several types of users in GitLab: ## User activity +GitLab tracks user contribution activity. You can follow or unfollow other users from their [user profiles](profile/index.md#access-your-user-profile). To view a user's activity in a top-level Activity view: @@ -92,6 +93,55 @@ To view a user's activity in a top-level Activity view: 1. In the GitLab menu, select **Activity**. 1. Select the **Followed users** tab. +### User contribution events + +Each of these contribution events is tracked: + +- `approved` + - Merge request +- `closed` + - [Epic](group/epics/index.md) + - Issue + - Merge request + - Milestone +- `commented` on any `Noteable` record. + - Alert + - Commit + - Design + - Issue + - Merge request + - Snippet +- `created` + - Design + - [Epic](group/epics/index.md) + - Issue + - Merge request + - Milestone + - Project + - Wiki page +- `destroyed` + - Design + - Milestone + - Wiki page +- `expired` + - Project membership +- `joined` + - Project membership +- `left` + - Project membership +- `merged` + - Merge request +- `pushed` commits to (or deleted commits from) a repository, individually or in bulk. + - Project +- `reopened` + - [Epic](group/epics/index.md) + - Issue + - Merge request + - Milestone +- `updated` + - Design + - Wiki page + ## Projects In GitLab, you can create [projects](project/index.md) to host diff --git a/doc/user/infrastructure/clusters/connect/index.md b/doc/user/infrastructure/clusters/connect/index.md index 636cb1bb457..21387998a17 100644 --- a/doc/user/infrastructure/clusters/connect/index.md +++ b/doc/user/infrastructure/clusters/connect/index.md @@ -6,62 +6,14 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Connect a cluster to GitLab **(FREE)** -You can create new or connect existing clusters to GitLab through different [levels](#cluster-levels), -using different [methods](#methods-to-connect-a-cluster-to-gitlab). +The [certificate-based Kubernetes integration with GitLab](../index.md) +was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) +in GitLab 14.5. To connect your clusters, use the [GitLab Kubernetes Agent](../../../clusters/agent/index.md). -Before getting started: - -1. Check the [supported Kubernetes cluster versions](#supported-cluster-versions). -1. Define the [cluster level](#cluster-levels) according to your case. - -After that: - -1. Choose the [method](#methods-to-connect-a-cluster-to-gitlab) -to connect your cluster according to your case. -1. [View your clusters](#view-your-clusters) connected to GitLab. - -## Methods to connect a cluster to GitLab - -GitLab offers three methods to connect existing and create new clusters: - -- **GitLab Kubernetes Agent**: the best solution to -[connect existing clusters](#connect-existing-clusters-to-gitlab). -- **Infrastructure as Code**: it's a broader infrastructure management -toolset that includes managing your cluster. It's the recommended -solution to [create a new cluster](#create-new-clusters-from-gitlab) -from GitLab. -- **Certificate-based method**: our first and legacy solution uses -cluster certificates to connect your cluster to GitLab. It is no longer -recommended for [security implications](#security-implications-for-clusters-connected-with-certificates). - -### Connect existing clusters to GitLab - -To safely connect and configure an existing cluster on the **project level**, -we **recommend** using the [GitLab Kubernetes Agent](../../../clusters/agent/index.md). -We are working to support [the Agent for connecting a cluster at the group level](https://gitlab.com/groups/gitlab-org/-/epics/5784). - -Alternatively, you can use [cluster certificates](../../../project/clusters/add_existing_cluster.md) -to connect clusters in all levels (projects, group, instance). However, -for [security implications](#security-implications-for-clusters-connected-with-certificates), -we don't recommend using this method. - -### Create new clusters from GitLab - -To safely create new clusters from GitLab, use -[Infrastructure as Code](../../iac/index.md#create-a-new-cluster-through-iac). - -The [certificate-based method to create a new cluster](../../../project/clusters/add_remove_clusters.md) -is still available through the GitLab UI but was **deprecated** in GitLab 14.0. -If possible, we don't recommend using this method. - -### Connect multiple clusters to a single project - -To connect multiple clusters to a single project in GitLab, -we **recommend** using the [GitLab Kubernetes Agent](../../../clusters/agent/index.md). - -You can also use the [certificate-based method](../../../project/clusters/multiple_kubernetes_clusters.md), -but, for [security implications](#security-implications-for-clusters-connected-with-certificates), -we don't recommend using this method. +<!-- TBA: (We need to resolve https://gitlab.com/gitlab-org/gitlab/-/issues/343660 before adding this line) +If you don't have a cluster yet, create one and connect it to GitLab through the Agent. +You can also create a new cluster from GitLab using [Infrastructure as Code](../../iac/index.md#create-a-new-cluster-through-iac). +--> ## Supported cluster versions @@ -85,7 +37,13 @@ Kubernetes version to any supported version at any time: Some GitLab features may support versions outside the range provided here. -## Cluster levels +## Cluster levels (DEPRECATED) + +> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +The [concept of cluster levels was deprecated](../index.md#cluster-levels) +in GitLab 14.5. Choose your cluster's level according to its purpose: @@ -118,6 +76,8 @@ your cluster's level. ## Security implications for clusters connected with certificates +> Connecting clusters to GitLab through cluster certificates was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + WARNING: The whole cluster security is based on a model where [developers](../../../permissions.md) are trusted, so **only trusted users should be allowed to control your clusters**. diff --git a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md index 3c934b72886..d1e3bd47b89 100644 --- a/doc/user/infrastructure/clusters/connect/new_gke_cluster.md +++ b/doc/user/infrastructure/clusters/connect/new_gke_cluster.md @@ -4,7 +4,16 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# New GKE cluster through IaC +# New GKE cluster through IaC (DEPRECATED) + +> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +The process described on this page uses cluster certificates to connect the +new cluster to GitLab, [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +You can still create a cluster and then connect it to GitLab through the [Agent](../index.md). +[An issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/343660) +to migrate this functionality to the [Agent](../index.md). Learn how to create a new cluster on Google Kubernetes Engine (GKE) through [Infrastructure as Code (IaC)](../../index.md). diff --git a/doc/user/infrastructure/clusters/index.md b/doc/user/infrastructure/clusters/index.md index 16ca6d02865..06a77912876 100644 --- a/doc/user/infrastructure/clusters/index.md +++ b/doc/user/infrastructure/clusters/index.md @@ -6,61 +6,68 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Kubernetes clusters **(FREE)** -> - Project-level clusters [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/35954) in GitLab 10.1. -> - Group-level clusters [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/34758) in GitLab 11.6. -> - Instance-level clusters [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39840) in GitLab 11.11. - -Kubernetes is a container orchestration platform to deploy applications -in a cluster without downtime and that scales as you need. - -With the GitLab integration with Kubernetes, you can: - -1. [Connect your cluster](#connect-your-cluster-to-gitlab). -1. [Manage your cluster](#manage-your-cluster). -1. [Deploy your cluster](#deploy-to-your-cluster). - -See the [Kubernetes clusters versions supported by GitLab](connect/index.md#supported-cluster-versions). - -## Connect your cluster to GitLab - -Learn how to [create new and connect existing clusters to GitLab](connect/index.md). - -## Manage your cluster - -- [Cluster Management Project](../../clusters/management_project.md): -create a project to manage your cluster's shared resources requiring -`cluster-admin` privileges such as an Ingress controller. - - [Cluster Management Project Template](../../clusters/management_project_template.md): start a cluster management project directly from a template. - - [Migrate to Cluster Management Project](../../clusters/migrating_from_gma_to_project_template.md): migrate from the deprecated GitLab Managed Apps to Cluster Management Projects. - - [GitLab Managed Apps](../../clusters/applications.md) (deprecated in favor of Cluster Management Projects): configure applications in your cluster directly from GitLab. -- [Cluster integrations](../../clusters/integrations.md): install -third-party applications into your cluster and manage them from GitLab. -- [GitLab-managed clusters](../../project/clusters/gitlab_managed_clusters.md): -enable GitLab to automatically create resources for your clusters. -- [Cost management](../../clusters/cost_management.md): see insights into your cluster's resource usage. -- [Crossplane integration](../../clusters/crossplane.md): manage your cluster's resources and cloud infrastructure with Crossplane. - -### Monitor your cluster - -- [Prometheus monitoring](../../project/integrations/prometheus_library/kubernetes.md): detect and monitor Kubernetes metrics with Prometheus. -- [NGINX monitoring](../../project/integrations/prometheus_library/nginx.md): automatically monitor NGINX Ingress. -- [Clusters health](manage/clusters_health.md): monitor your cluster's health, such as CPU and memory usage. - -### Secure your cluster - -- [Container Host Security](../../project/clusters/protect/container_host_security/index.md): monitor and block activity inside a container and enforce security policies across the cluster. -- [Container Network security](../../project/clusters/protect/container_network_security/index.md): filter traffic going in and out of the cluster and traffic between pods through a firewall with Cilium NetworkPolicies. - -## Deploy to your cluster - -- [CI/CD Tunnel](../../clusters/agent/ci_cd_tunnel.md): use the CI/CD Tunnel to run Kubernetes commands from different projects. -- [Inventory object](deploy/inventory_object.md): track objects applied to a cluster configured with the Kubernetes Agent. -- [Auto DevOps](../../../topics/autodevops/index.md): enable Auto DevOps -to allow GitLab automatically detect, build, test, and deploy applications. -- [Cluster environments](../../clusters/environments.md): view CI/CD environments deployed to Kubernetes clusters. -- [Canary Deployments](../../project/canary_deployments.md): deploy app updates to a small portion of the fleet with this Continuous Delivery strategy. -- [Deploy to your cluster](../../project/clusters/deploy_to_cluster.md): -deploy applications into your cluster using cluster certificates. -- [Deploy Boards](../../project/deploy_boards.md): view the current health and status of each CI/CD environment running on your cluster, and the status of deployment pods. -- [Pod logs](../../project/clusters/kubernetes_pod_logs.md): view the logs of your cluster's running pods. -- [Serverless](../../project/clusters/serverless/index.md) (deprecated): deploy Serverless applications in Kubernetes environments and cloud Function as a Service (FaaS) environments. +To connect clusters to GitLab, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). + +## Certificate-based Kubernetes integration (DEPRECATED) + +WARNING: +In GitLab 14.5, the certificate-based method to connect Kubernetes clusters +to GitLab was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8), +as well as its related [features](#deprecated-features). + +The certificate-based Kubernetes integration with GitLab is deprecated. +It had the following issues: + +- There were security issues as it required direct access to the Kube API by GitLab. +- The configuration options weren't flexible. +- The integration was flaky. +- Users were constantly reporting issues with features based on this model. + +For this reason, we started to build features based on a new model, the +[GitLab Kubernetes Agent](../../clusters/agent/index.md). +Maintaining both methods in parallel caused a lot of confusion +and significantly increased the complexity to use, develop, maintain, and +document them. For this reason, we decided to deprecate them to focus on the +new model. + +Certificate-based features will continue to receive security and critical +fixes, and features built on top of it will continue to work with the supported +Kubernetes versions. The removal of these features from GitLab is not +scheduled yet. +Follow this [epic](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) +for updates. + +You can find technical information about why we moved away from cluster certificates into +the Kubernetes Agent model on the [Agent's blueprint documentation](../../../architecture/blueprints/gitlab_to_kubernetes_communication/index.md). + +## Deprecated features + +- [Create a new cluster through cluster certificates](../../project/clusters/add_remove_clusters.md) +- [Connect an existing cluster through cluster certificates](../../project/clusters/add_existing_cluster.md) +- [Access controls](../../project/clusters/cluster_access.md) +- [GitLab-managed clusters](../../project/clusters/gitlab_managed_clusters.md) +- [GitLab Managed Apps](../../clusters/applications.md) +- [Deploy applications through certificate-based connection](../../project/clusters/deploy_to_cluster.md) +- [Cluster Management Project](../../clusters/management_project.md) +- [Cluster integrations](../../clusters/integrations.md) +- [Cluster cost management](../../clusters/cost_management.md) +- [Cluster environments](../../clusters/environments.md) +- [Canary Deployments](../../project/canary_deployments.md) +- [Serverless](../../project/clusters/serverless/index.md) +- [Deploy Boards](../../project/deploy_boards.md) +- [Pod logs](../../project/clusters/kubernetes_pod_logs.md) +- [Clusters health](manage/clusters_health.md) +- [Crossplane integration](../../clusters/crossplane.md) +- [Auto Deploy](../../../topics/autodevops/stages.md#auto-deploy) +- [Web terminals](../../../administration/integration/terminal.md) + +### Cluster levels + +The concept of [project-level](../../project/clusters/index.md), +[group-level](../../group/clusters/index.md), and +[instance-level](../../instance/clusters/index.md) clusters becomes +extinct in the new model, although the functionality remains to some extent. + +The Agent is always configured in a single GitLab project, but you can use the CI/CD Tunnel to +[authorize other projects and groups to use the same Agent](../../clusters/agent/repository.md#authorize-projects-and-groups-to-use-an-agent). +By doing so, you are granting these projects and groups access to the same cluster, which is similar to group-level clusters' use case. diff --git a/doc/user/infrastructure/clusters/manage/clusters_health.md b/doc/user/infrastructure/clusters/manage/clusters_health.md index 009945589ad..eeb931f392f 100644 --- a/doc/user/infrastructure/clusters/manage/clusters_health.md +++ b/doc/user/infrastructure/clusters/manage/clusters_health.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Clusters health **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4701) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 10.6. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/208224) to GitLab Free in 13.2. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4701) in GitLab 10.6. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/208224) from GitLab Ultimate to GitLab Free in 13.2. When [the Prometheus cluster integration is enabled](../../../clusters/integrations.md#prometheus-cluster-integration), GitLab monitors the cluster's health. At the top of the cluster settings page, CPU and Memory utilization is displayed, along with the total amount available. Keeping an eye on cluster resources can be important, if the cluster runs out of memory pods may be shutdown or fail to start. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/apparmor.md b/doc/user/infrastructure/clusters/manage/management_project_applications/apparmor.md index 7fbbbac866c..ae335a180e8 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/apparmor.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/apparmor.md @@ -4,7 +4,7 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install AppArmor with a cluster management project +# Install AppArmor with a cluster management project **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/certmanager.md b/doc/user/infrastructure/clusters/manage/management_project_applications/certmanager.md index 9ef7bd0f3ff..12f99af8d8d 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/certmanager.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/certmanager.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install cert-manager with a cluster management project +# Install cert-manager with a cluster management project **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. > - Support for cert-manager v1.4 was [introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/69405) in GitLab 14.3. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/cilium.md b/doc/user/infrastructure/clusters/manage/management_project_applications/cilium.md index c19bfbfb1b1..b5959624954 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/cilium.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/cilium.md @@ -4,7 +4,7 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install Cilium with a cluster management project +# Install Cilium with a cluster management project **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/elasticstack.md b/doc/user/infrastructure/clusters/manage/management_project_applications/elasticstack.md index dbde9bd90b0..3bd675b7439 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/elasticstack.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/elasticstack.md @@ -4,7 +4,7 @@ group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install Elastic Stack with a cluster management project +# Install Elastic Stack with a cluster management project **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/falco.md b/doc/user/infrastructure/clusters/manage/management_project_applications/falco.md index 7bd2a4a5133..50401e9a391 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/falco.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/falco.md @@ -4,7 +4,7 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install Falco with a cluster management project +# Install Falco with a cluster management project **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/fluentd.md b/doc/user/infrastructure/clusters/manage/management_project_applications/fluentd.md index c5de0511c2f..ea3a3503f9b 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/fluentd.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/fluentd.md @@ -4,7 +4,7 @@ group: Container Security info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install Fluentd with a cluster management project +# Install Fluentd with a cluster management project **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/ingress.md b/doc/user/infrastructure/clusters/manage/management_project_applications/ingress.md index 5ee26db754e..503f077df14 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/ingress.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/ingress.md @@ -4,7 +4,7 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install Ingress with a cluster management project +# Install Ingress with a cluster management project **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md b/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md index 3420f340c94..fd2eed25997 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/prometheus.md @@ -4,7 +4,7 @@ group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install Prometheus with a cluster management project +# Install Prometheus with a cluster management project **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md b/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md index 300350010af..9e5d7860a67 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/sentry.md @@ -4,7 +4,7 @@ group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install Sentry with a cluster management project +# Install Sentry with a cluster management project **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. diff --git a/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md b/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md index d6b4eb5c157..4618a95f986 100644 --- a/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md +++ b/doc/user/infrastructure/clusters/manage/management_project_applications/vault.md @@ -1,10 +1,10 @@ --- -stage: Release -group: Release +stage: Configure +group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Install Vault with a cluster management project +# Install Vault with a cluster management project **(FREE)** > [Introduced](https://gitlab.com/gitlab-org/project-templates/cluster-management/-/merge_requests/5) in GitLab 14.0. diff --git a/doc/user/infrastructure/iac/mr_integration.md b/doc/user/infrastructure/iac/mr_integration.md index 853a39a59a8..e92b2d919ae 100644 --- a/doc/user/infrastructure/iac/mr_integration.md +++ b/doc/user/infrastructure/iac/mr_integration.md @@ -16,7 +16,7 @@ enabling you to see statistics about the resources that Terraform creates, modifies, or destroys. WARNING: -Like any other job artifact, Terraform Plan data is [viewable by anyone with Guest access](../../permissions.md) to the repository. +Like any other job artifact, Terraform Plan data is viewable by anyone with the Guest [role](../../permissions.md) on the repository. Neither Terraform nor GitLab encrypts the plan file by default. If your Terraform Plan includes sensitive data such as passwords, access tokens, or certificates, we strongly recommend encrypting plan output or modifying the project visibility settings. @@ -77,8 +77,7 @@ To manually configure a GitLab Terraform Report artifact: terraform: $PLAN_JSON ``` - For a full example using the pre-built image, see [Example `.gitlab-ci.yml` - file](#example-gitlab-ciyml-file). + For a full example using the pre-built image, see [Example `.gitlab-ci.yml` file](#example-gitlab-ciyml-file). For an example displaying multiple reports, see [`.gitlab-ci.yml` multiple reports file](#multiple-terraform-plan-reports). diff --git a/doc/user/infrastructure/iac/terraform_state.md b/doc/user/infrastructure/iac/terraform_state.md index fb051c7fa14..84d1edbe2f7 100644 --- a/doc/user/infrastructure/iac/terraform_state.md +++ b/doc/user/infrastructure/iac/terraform_state.md @@ -205,7 +205,7 @@ and the CI YAML file: The output from the above `terraform` commands should be viewable in the job logs. WARNING: -Like any other job artifact, Terraform plan data is [viewable by anyone with Guest access](../../permissions.md) to the repository. +Like any other job artifact, Terraform plan data is viewable by anyone with the Guest [role](../../permissions.md) on the repository. Neither Terraform nor GitLab encrypts the plan file by default. If your Terraform plan includes sensitive data such as passwords, access tokens, or certificates, GitLab strongly recommends encrypting plan output or modifying the project visibility settings. diff --git a/doc/user/infrastructure/index.md b/doc/user/infrastructure/index.md index e99dc691774..3bb518596cc 100644 --- a/doc/user/infrastructure/index.md +++ b/doc/user/infrastructure/index.md @@ -29,13 +29,12 @@ Learn more about how GitLab can help you run [Infrastructure as Code](iac/index. ## Integrated Kubernetes management -GitLab has special integrations with Kubernetes to help you deploy, manage and troubleshoot -third-party or custom applications in Kubernetes clusters. Auto DevOps provides a full -DevSecOps pipeline by default targeted at Kubernetes based deployments. To support -all the GitLab features, GitLab offers a cluster management project for easy onboarding. -The deploy boards provide quick insights into your cluster, including pod logs tailing. +The GitLab integration with Kubernetes helps you to install, configure, manage, deploy, and troubleshoot +cluster applications. With the GitLab Kubernetes Agent, you can connect clusters behind a firewall, +have real-time access to API endpoints, perform pull-based or push-based deployments for production +and non-production environments, and much more. -Learn more about the [GitLab integration with Kubernetes](clusters/index.md). +Learn more about the [GitLab Kubernetes Agent](../clusters/agent/index.md). ## Runbooks in GitLab diff --git a/doc/user/instance/clusters/index.md b/doc/user/instance/clusters/index.md index de09cd4845e..4bbd82d01a8 100644 --- a/doc/user/instance/clusters/index.md +++ b/doc/user/instance/clusters/index.md @@ -4,9 +4,14 @@ group: unassigned info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Instance-level Kubernetes clusters **(FREE SELF)** +# Instance-level Kubernetes clusters (certificate-based) (DEPRECATED) **(FREE SELF)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39840) in GitLab 11.11. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/39840) in GitLab 11.11. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. To connect clusters to GitLab, +use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). Similar to [project-level](../../project/clusters/index.md) and [group-level](../../group/clusters/index.md) Kubernetes clusters, diff --git a/doc/user/markdown.md b/doc/user/markdown.md index 5e600b6e0d1..d20e9c7a30e 100644 --- a/doc/user/markdown.md +++ b/doc/user/markdown.md @@ -125,8 +125,6 @@ You can also use [Kroki](https://kroki.io) to create a wide variety of diagrams. #### Mermaid -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/15107) in GitLab 10.3. - Visit the [official page](https://mermaidjs.github.io/) for more details. The [Mermaid Live Editor](https://mermaid-js.github.io/mermaid-live-editor/) helps you learn Mermaid and debug issues in your Mermaid code. Use it to identify and resolve @@ -211,20 +209,20 @@ Sometimes you want to :monkey: around a bit and add some :star2: to your :speech You can use it to point out a :bug: or warn about :speak_no_evil: patches. And if someone improves your really :snail: code, send them some :birthday:. People :heart: you for that. -If you're new to this, don't be :fearful:. You can join the emoji :family:. All you need to do is to look up one of the supported codes. +If you're new to this, don't be :fearful:. You can join the emoji :family:. Just look up one of the supported codes. Consult the [Emoji Cheat Sheet](https://www.emojicopy.com) for a list of all supported emoji codes. :thumbsup: ``` -Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/monkey.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/star2.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">. Well we have a gift for you: +Sometimes you want to <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/monkey.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> around a bit and add some <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/star2.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> to your <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/speech_balloon.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">. Well we have a gift for you: -<img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/zap.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">You can use emoji anywhere GitLab Flavored Markdown is supported. <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/v.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> +<img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/zap.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">You can use emoji anywhere GitLab Flavored Markdown is supported. <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/v.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> -You can use it to point out a<img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/bug.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/speak_no_evil.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> patches. If someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/snail.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/birthday.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">. People <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/heart.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> you for that. +You can use it to point out a<img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/bug.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> or warn about <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/speak_no_evil.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> patches. If someone improves your really <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/snail.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> code, send them some <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/birthday.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">. People <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/heart.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> you for that. -If you're new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/fearful.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">. You can join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/family.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">. All you need to do is to look up one of the supported codes. +If you're new to this, don't be <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/fearful.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">. You can join the emoji <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/family.png" width="20px" height="20px" style="display:inline;margin:0;border: 0">. Just look up one of the supported codes. -Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/app/assets/images/emoji/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> +Consult the [Emoji Cheat Sheet](https://www.webfx.com/tools/emoji-cheat-sheet/) for a list of all supported emoji codes. <img src="https://gitlab.com/gitlab-org/gitlab-foss/raw/master/public/-/emojis/2/thumbsup.png" width="20px" height="20px" style="display:inline;margin:0;border: 0"> #### Emojis and your operating system @@ -236,7 +234,7 @@ emojis where there is no support. <!-- vale gitlab.Spelling = NO --> -On Linux, you can download [Noto Color Emoji](https://www.google.com/get/noto/help/emoji/) +On Linux, you can download [Noto Color Emoji](https://github.com/googlefonts/noto-emoji) to get full native emoji support. Ubuntu 18.04 (like many modern Linux distributions) has this font installed by default. @@ -244,8 +242,6 @@ this font installed by default. ### Front matter -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/23331) in GitLab 11.6. - Front matter is metadata included at the beginning of a Markdown document, preceding the content. This data can be used by static site generators like [Jekyll](https://jekyllrb.com/docs/front-matter/), [Hugo](https://gohugo.io/content-management/front-matter/), and many other applications. @@ -356,18 +352,18 @@ in a [code block](#code-spans-and-blocks) with the language declared as `math` i on a separate line: ````markdown -This math is inline $`a^2+b^2=c^2`$. +This math is inline: $`a^2+b^2=c^2`$. -This is on a separate line: +This math is on a separate line: ```math a^2+b^2=c^2 ``` ```` -This math is inline $`a^2+b^2=c^2`$. +This math is inline: $`a^2+b^2=c^2`$. -This is on a separate line +This math is on a separate line: ```math a^2+b^2=c^2 @@ -408,16 +404,17 @@ To create a task list, follow the format of an ordered or unordered list: ### Table of contents A table of contents is an unordered list that links to subheadings in the document. - -To add a table of contents to a Markdown file, wiki page, issue request, or merge request -description, add either the `[[_TOC_]]` or `[TOC]` tag on its own line. - -NOTE: You can add a table of contents to issues and merge requests, but you can't add one -to notes or comments. +to notes or comments. Add either the `[[_TOC_]]` or `[TOC]` tag on its own line +to the **Description** field of any of the supported content types: + +- Markdown files. +- Wiki pages. +- Issues. +- Merge requests. ```markdown -This is an intro sentence to my Wiki page. +This sentence introduces my wiki page. [[_TOC_]] @@ -579,7 +576,7 @@ by starting the lines of the blockquote with `>`: Quote break. -> This is a very long line that is still quoted properly when it wraps. Keep writing to make sure this line is long enough to actually wrap for everyone. You can also *add* **Markdown** into a blockquote. +> This very long line is still quoted properly when it wraps. Keep writing to make sure this line is long enough to actually wrap for everyone. You can also *add* **Markdown** into a blockquote. ``` > Blockquotes help you emulate reply text. @@ -587,7 +584,7 @@ Quote break. Quote break. -> This is a very long line that is still quoted properly when it wraps. Keep writing to make sure this line is long enough to actually wrap for everyone. You can also *add* **Markdown** into a blockquote. +> This very long line is still quoted properly when it wraps. Keep writing to make sure this line is long enough to actually wrap for everyone. You can also *add* **Markdown** into a blockquote. #### Multiline blockquote @@ -706,7 +703,7 @@ puts markdown.to_html ``` No language indicated, so no syntax highlighting. -s = "There is no highlighting for this." +s = "No highlighting is shown for this line." But let's throw in a <b>tag</b>. ``` ```` @@ -733,13 +730,13 @@ puts markdown.to_html ```plaintext No language indicated, so no syntax highlighting. -s = "There is no highlighting for this." +s = "No highlighting is shown for this line." But let's throw in a <b>tag</b>. ``` ### Emphasis -There are multiple ways to emphasize text in Markdown. You can italicize, bold, strikethrough, +In Markdown, you can emphasize text in multiple ways. You can italicize, bold, strikethrough, and combine these emphasis styles together. Strikethrough is not part of the core Markdown standard, but is part of GitLab Flavored Markdown. @@ -811,10 +808,6 @@ the note content. Regardless of the tag names, the relative order of the reference tags determines the rendered numbering. -Reference tags can use letters and other characters. Avoid using lowercase `w` or an underscore -(`_`) in footnote tag names until [this bug](https://gitlab.com/gitlab-org/gitlab/-/issues/24423) is -resolved. - <!-- Do not edit the following codeblock. It uses HTML to skip the Vale ReferenceLinks test. --> @@ -823,9 +816,9 @@ Do not edit the following codeblock. It uses HTML to skip the Vale ReferenceLink This reference tag is a mix of letters and numbers. [^footnote-42] -[^1]: This is the text inside a footnote. +[^1]: This text is inside a footnote. -[^footnote-42]: This is another footnote. +[^footnote-42]: This text is another footnote. </code></pre> A footnote reference tag looks like this:[^1] @@ -837,9 +830,9 @@ Do not delete the single space before the [^1] and [^footnotes] references below These are used to force the Vale ReferenceLinks check to skip these examples. --> - [^1]: This is the text inside a footnote. + [^1]: This text is inside a footnote. - [^footnote-42]: This is another footnote. + [^footnote-42]: This text is another footnote. ### Headers @@ -897,8 +890,8 @@ Would generate the following link IDs: 1. `this-header-has-spaces-in-it-2` 1. `this-header-has-3-5-in-it-and-parentheses` -Note that the emoji processing happens before the header IDs are generated, so the -emoji is converted to an image which is then removed from the ID. +Emoji processing happens before the header IDs are generated. The +emoji is converted to an image, which is then removed from the ID. ### Horizontal Rule @@ -938,7 +931,7 @@ Reference-style (hover to see title text): </code></pre> <!-- -DO NOT change the name of markdown_logo.png. This is used for a test in +DO NOT change the name of markdown_logo.png. This file is used for a test in spec/controllers/help_controller_spec.rb. --> @@ -955,7 +948,7 @@ Do not change to a reference style link.  -In the rare case where you need to set a specific height or width for an image, +In the rare case where you must set a specific height or width for an image, you can use the `img` HTML tag instead of Markdown and set its `height` and `width` parameters. @@ -1133,8 +1126,8 @@ These details <em>remain</em> <b>hidden</b> until expanded. ### Line breaks A line break is inserted (a new paragraph starts) if the previous text is -ended with two newlines, like when you hit <kbd>Enter</kbd> twice in a row. If you only -use one newline (hit <kbd>Enter</kbd> once), the next sentence remains part of the +ended with two newlines, like when you press <kbd>Enter</kbd> twice in a row. If you only +use one newline (select <kbd>Enter</kbd> once), the next sentence remains part of the same paragraph. Use this approach if you want to keep long lines from wrapping, and keep them editable: @@ -1160,7 +1153,8 @@ in the *same paragraph*. #### Newlines -GitLab Flavored Markdown adheres to the Markdown specification in how [paragraphs and line breaks are handled](https://spec.commonmark.org/current/). +GitLab Flavored Markdown adheres to the Markdown specification for handling +[paragraphs and line breaks](https://spec.commonmark.org/current/). A paragraph is one or more consecutive lines of text, separated by one or more blank lines (two newlines at the end of the first paragraph), as [explained above](#line-breaks). @@ -1182,25 +1176,25 @@ A new line due to the previous backslash. ### Links -There are two ways to create links, inline-style and reference-style: +You can create links two ways: inline-style and reference-style. For example: <!-- Do not edit the following codeblock. It uses HTML to skip the Vale ReferenceLinks test. --> -<pre class="highlight"><code>- This is an [inline-style link](https://www.google.com) -- This is a [link to a repository file in the same directory](index.md) -- This is a [relative link to a readme one directory higher](../index.md) -- This is a [link that also has title text](https://www.google.com "This link takes you to Google!") +<pre class="highlight"><code>- This line shows an [inline-style link](https://www.google.com) +- This line shows a [link to a repository file in the same directory](index.md) +- This line shows a [relative link to a readme one directory higher](../index.md) +- This line shows a [link that also has title text](https://www.google.com "This link takes you to Google!") Using header ID anchors: -- This links to [a section on a different Markdown page, using a "#" and the header ID](index.md#overview) -- This links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) +- This line links to [a section on a different Markdown page, using a "#" and the header ID](index.md#overview) +- This line links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) Using references: -- This is a [reference-style link, see below][Arbitrary case-insensitive reference text] +- This line shows a [reference-style link, see below][Arbitrary case-insensitive reference text] - You can [use numbers for reference-style link definitions, see below][1] - Or leave it empty and use the [link text itself][], see below. @@ -1211,15 +1205,15 @@ Some text to show that the reference links can follow later. [link text itself]: https://www.reddit.com </code></pre> -- This is an [inline-style link](https://www.google.com) -- This is a [link to a repository file in the same directory](index.md) -- This is a [relative link to a README one directory higher](../index.md) -- This is a [link that also has title text](https://www.google.com "This link takes you to Google!") +- This line shows an [inline-style link](https://www.google.com) +- This line shows a [link to a repository file in the same directory](index.md) +- This line shows a [relative link to a README one directory higher](../index.md) +- This line shows a [link that also has title text](https://www.google.com "This link takes you to Google!") Using header ID anchors: -- This links to [a section on a different Markdown page, using a "#" and the header ID](index.md#overview) -- This links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) +- This line links to [a section on a different Markdown page, using a "#" and the header ID](index.md#overview) +- This line links to [a different section on the same page, using a "#" and the header ID](#header-ids-and-links) Using references: @@ -1228,7 +1222,7 @@ The example below uses in-line links to pass the Vale ReferenceLinks test. Do not change to reference style links. --> -- This is a [reference-style link, see below](https://www.mozilla.org/en-US/) +- This line is a [reference-style link, see below](https://www.mozilla.org/en-US/) - You can [use numbers for reference-style link definitions, see below](https://slashdot.org) - Or leave it empty and use the [link text itself](https://www.reddit.com), see below. @@ -1236,7 +1230,7 @@ Some text to show that the reference links can follow later. NOTE: Relative links do not allow the referencing of project files in a wiki -page, or a wiki page in a project file. The reason for this is that a wiki is always +page, or a wiki page in a project file. The reason: a wiki is always in a separate Git repository in GitLab. For example, `[I'm a reference-style link](style)` points the link to `wikis/style` only when the link is inside of a wiki Markdown file. @@ -1265,14 +1259,14 @@ GitLab Flavored Markdown auto-links almost any URL you put into your text: <!-- vale gitlab.Spelling = YES --> ### Lists -Ordered and unordered lists can be created. +You can create ordered and unordered lists. For an ordered list, add the number you want the list to start with, like `1.`, followed by a space, at the start of each line for ordered lists. -After the first number, it does not matter what number you use, ordered lists are +After the first number, it does not matter what number you use. Ordered lists are numbered automatically by vertical order, so repeating `1.` for all items in the same list is common. If you start with a number other than `1.`, it uses that as the first -number, and count up from there. +number, and counts up from there. Examples: @@ -1364,10 +1358,9 @@ Example: --- -If the paragraph of the first item is not indented with the proper number of spaces, +If the first item's paragraph isn't indented with the proper number of spaces, the paragraph appears outside the list, instead of properly indented under the list item. - -Example: +For example: ```markdown 1. First ordered list item @@ -1459,13 +1452,13 @@ use `<br>` tags to force a cell to have multiple lines: ```markdown | Name | Details | | --- | --- | -| Item1 | This is on one line | +| Item1 | This text is on one line | | Item2 | This item has:<br>- Multiple items<br>- That we want listed separately | ``` | Name | Details | | --- | --- | -| Item1 | This is on one line | +| Item1 | This text is on one line | | Item2 | This item has:<br>- Multiple items<br>- That we want listed separately | You can use HTML formatting in GitLab itself to add [task lists](#task-lists) with checkboxes, diff --git a/doc/user/packages/composer_repository/index.md b/doc/user/packages/composer_repository/index.md index 6e3af45df17..7861258e23f 100644 --- a/doc/user/packages/composer_repository/index.md +++ b/doc/user/packages/composer_repository/index.md @@ -6,9 +6,9 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Composer packages in the Package Registry **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15886) in GitLab Premium 13.2. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. -> - Support for Composer 2.0 [added](https://gitlab.com/gitlab-org/gitlab/-/issues/259840) in GitLab Free 13.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15886) in GitLab 13.2. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. +> - Support for Composer 2.0 [added](https://gitlab.com/gitlab-org/gitlab/-/issues/259840) in GitLab 13.10. WARNING: The Composer package registry for GitLab is under development and isn't ready for production use due to @@ -149,7 +149,7 @@ Do not save unless you want to overwrite the existing CI/CD file. When you publish: - The same package with different data, it overwrites the existing package. -- The same package with the same data, a `404 Bad request` error occurs. +- The same package with the same data, a `400 Bad request` error occurs. ## Install a Composer package diff --git a/doc/user/packages/conan_repository/index.md b/doc/user/packages/conan_repository/index.md index 33c48478921..0f32f68d250 100644 --- a/doc/user/packages/conan_repository/index.md +++ b/doc/user/packages/conan_repository/index.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Conan packages in the Package Registry **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8248) in GitLab Premium 12.6. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/8248) in GitLab 12.6. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. WARNING: The Conan package registry for GitLab is under development and isn't ready for production use due to @@ -265,7 +265,8 @@ conan upload Hello/0.1@mycompany/beta --all ## Publish a Conan package by using CI/CD -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11678) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11678) in GitLab 12.7. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. To work with Conan commands in [GitLab CI/CD](../../../ci/index.md), you can use `CI_JOB_TOKEN` in place of the personal access token in your commands. diff --git a/doc/user/packages/container_registry/index.md b/doc/user/packages/container_registry/index.md index c39552c1edb..c9cdc8643f4 100644 --- a/doc/user/packages/container_registry/index.md +++ b/doc/user/packages/container_registry/index.md @@ -157,7 +157,7 @@ To use CI/CD to authenticate, you can use: - A [CI job token](../../../ci/jobs/ci_job_token.md). ```shell - docker login -u $CI_JOB_USER -p $CI_JOB_TOKEN $CI_REGISTRY + docker login -u $CI_REGISTRY_USER -p $CI_JOB_TOKEN $CI_REGISTRY ``` - A [deploy token](../../project/deploy_tokens/index.md#gitlab-deploy-token) with the minimum scope of: @@ -309,7 +309,7 @@ in addition to the steps in the [Docker-in-Docker](../../../ci/docker/using_docker_build.md#use-the-docker-executor-with-the-docker-image-docker-in-docker) section: 1. Update the `image` and `service` to point to your registry. -1. Add a service [alias](../../../ci/yaml/index.md#servicesalias). +1. Add a service [alias](../../../ci/services/index.md#available-settings-for-services). Below is an example of what your `.gitlab-ci.yml` should look like: @@ -339,7 +339,7 @@ in addition to the steps in the [Docker-in-Docker](../../../ci/docker/using_docker_build.md#use-the-docker-executor-with-the-docker-image-docker-in-docker) section: 1. Update the `image` and `service` to point to your registry. -1. Add a service [alias](../../../ci/yaml/index.md#servicesalias). +1. Add a service [alias](../../../ci/services/index.md#available-settings-for-services). Below is an example of what your `.gitlab-ci.yml` should look like: diff --git a/doc/user/packages/debian_repository/index.md b/doc/user/packages/debian_repository/index.md index 36d85d94161..89427174dcd 100644 --- a/doc/user/packages/debian_repository/index.md +++ b/doc/user/packages/debian_repository/index.md @@ -79,10 +79,10 @@ packages on the group level, create a distribution with the same `codename`. To create a project-level distribution: ```shell -curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<project_id>/debian_distributions?codename=unstable" +curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/<project_id>/debian_distributions?codename=<codename>" ``` -Example response: +Example response with `codename=unstable`: ```json { @@ -146,10 +146,23 @@ To install a package: | sudo tee /etc/apt/auth.conf.d/gitlab_project.conf ``` + Download your distribution key: + + ```shell + sudo mkdir -p /usr/local/share/keyrings + curl --header "PRIVATE-TOKEN: <your_access_token>" \ + "https://gitlab.example.com/api/v4/projects/<project_id>/debian_distributions/<codename>/key.asc" \ + | \ + gpg --dearmor \ + | \ + sudo tee /usr/local/share/keyrings/<codename>-archive-keyring.gpg \ + > /dev/null + ``` + Add your project as a source: ```shell - echo 'deb [trusted=yes] https://gitlab.example.com/api/v4/projects/<project_id>/packages/debian <codename> <component1> <component2>' \ + echo 'deb [ signed-by=/usr/local/share/keyrings/<codename>-archive-keyring.gpg ] https://gitlab.example.com/api/v4/projects/<project_id>/packages/debian <codename> <component1> <component2>' \ | sudo tee /etc/apt/sources.list.d/gitlab_project.list sudo apt-get update ``` diff --git a/doc/user/packages/dependency_proxy/index.md b/doc/user/packages/dependency_proxy/index.md index 5d482a15460..fbd1cb84580 100644 --- a/doc/user/packages/dependency_proxy/index.md +++ b/doc/user/packages/dependency_proxy/index.md @@ -6,11 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Dependency Proxy **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.11. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) to GitLab Free in GitLab 13.6. -> - [Support for private groups](https://gitlab.com/gitlab-org/gitlab/-/issues/11582) in GitLab Free 13.7. -> - Anonymous access to images in public groups is no longer available starting in GitLab Free 13.7. -> - [Support for pull-by-digest and Docker version 20.x](https://gitlab.com/gitlab-org/gitlab/-/issues/290944) in GitLab Free 13.10. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/7934) in GitLab 11.11. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/273655) from GitLab Premium to GitLab Free in 13.6. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11582) support for private groups in GitLab 13.7. +> - Anonymous access to images in public groups is no longer available starting in GitLab 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/290944) support for pull-by-digest and Docker version 20.x in GitLab 13.10. The GitLab Dependency Proxy is a local proxy you can use for your frequently-accessed upstream images. @@ -59,7 +59,7 @@ Prerequisites: ### Authenticate with the Dependency Proxy -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11582) in GitLab Free 13.7. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11582) in GitLab 13.7. > - It's [deployed behind a feature flag](../../feature_flags.md), enabled by default. > - It's enabled on GitLab.com. > - It's recommended for production use. @@ -98,7 +98,7 @@ Proxy. #### Authenticate within CI/CD > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/280582) in GitLab 13.7. -> - Automatic runner authentication [added](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27302) in GitLab 13.9. +> - Automatic runner authentication, when using the Dependency Proxy to pull the image for the job, was [added](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/27302) in GitLab 13.9. > - The prefix for group names containing uppercase letters was [fixed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/54559) in GitLab 13.10. Runners log in to the Dependency Proxy automatically. To pull through @@ -134,6 +134,32 @@ Proxy manually without including the port: docker pull gitlab.example.com:443/my-group/dependency_proxy/containers/alpine:latest ``` +Example when using the Dependency Proxy to build an image: + +```plaintext +# Dockerfile +FROM gitlab.example.com:443/my-group/dependency_proxy/containers/alpine:latest +``` + +```yaml +# .gitlab-ci.yml +image: docker:19.03.12 + +variables: + DOCKER_HOST: tcp://docker:2375 + DOCKER_TLS_CERTDIR: "" + +services: + - docker:19.03.12-dind + +build: + image: docker:19.03.12 + before_script: + - docker login -u $CI_DEPENDENCY_PROXY_USER -p $CI_DEPENDENCY_PROXY_PASSWORD $CI_DEPENDENCY_PROXY_SERVER + script: + - docker build -t test . +``` + You can also use [custom CI/CD variables](../../../ci/variables/index.md#custom-cicd-variables) to store and access your personal access token or deploy token. ### Store a Docker image in Dependency Proxy cache @@ -189,7 +215,7 @@ If you clear the cache, the next time a pipeline runs it must pull an image or t ### Cleanup policies -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294187) in GitLab Free 14.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/294187) in GitLab 14.4. The cleanup policy is a scheduled job you can use to clear cached images that are no longer used, freeing up additional storage space. The policies use time-to-live (TTL) logic: @@ -229,7 +255,7 @@ files are ignored and new files are downloaded and cached from the external regi ## Docker Hub rate limits and the Dependency Proxy -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241639) in GitLab Free 13.7. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/241639) in GitLab 13.7. <i class="fa fa-youtube-play youtube" aria-hidden="true"></i> Watch how to [use the Dependency Proxy to help avoid Docker Hub rate limits](https://youtu.be/Nc4nUo7Pq08). diff --git a/doc/user/packages/generic_packages/index.md b/doc/user/packages/generic_packages/index.md index 6d35273ae6f..5b7316a495e 100644 --- a/doc/user/packages/generic_packages/index.md +++ b/doc/user/packages/generic_packages/index.md @@ -35,7 +35,10 @@ When you publish a package file, if the package does not exist, it is created. Prerequisites: -- You need to [authenticate with the API](../../../api/index.md#authentication). If authenticating with a deploy token, it must be configured with the `write_package_registry` scope. +- You must [authenticate with the API](../../../api/index.md#authentication). + If authenticating with a deploy token, it must be configured with the `write_package_registry` + scope. If authenticating with a personal access token or project access token, it must be + configured with the `api` scope. ```plaintext PUT /projects/:id/packages/generic/:package_name/:package_version/:file_name?status=:status @@ -48,6 +51,7 @@ PUT /projects/:id/packages/generic/:package_name/:package_version/:file_name?sta | `package_version` | string | yes | The package version. The following regex validates this: `\A(\.?[\w\+-]+\.?)+\z`. You can test your version strings on [Rubular](https://rubular.com/r/aNCV0wG5K14uq8). | `file_name` | string | yes | The filename. It can contain only lowercase letters (`a-z`), uppercase letter (`A-Z`), numbers (`0-9`), dots (`.`), hyphens (`-`), or underscores (`_`). | `status` | string | no | The package status. It can be `default` (default) or `hidden`. Hidden packages do not appear in the UI or [package API list endpoints](../../../api/packages.md). +| `select` | string | no | The response payload. By default, the response is empty. Valid values are: `package_file`. `package_file` returns details of the package file record created by this request. Provide the file context in the request body. @@ -59,7 +63,7 @@ curl --header "PRIVATE-TOKEN: <your_access_token>" \ "https://gitlab.example.com/api/v4/projects/24/packages/generic/my_package/0.0.1/file.txt?status=hidden" ``` -Example response: +Example response without attribute `select`: ```json { @@ -67,6 +71,34 @@ Example response: } ``` +Example response with attribute `select = package_file`: + +```json +{ + "id": 1, + "package_id": 1, + "created_at": "2021-10-12T12:05:23.387Z", + "updated_at": "2021-10-12T12:05:23.387Z", + "size": 0, + "file_store": 1, + "file_md5": null, + "file_sha1": null, + "file_name": "file.txt", + "file": { + "url": "/6b/86/6b86b273ff34fce19d6b804eff5a3f5747ada4eaa22f1d49c01e52ddb7875b4b/packages/26/files/36/file.txt" + }, + "file_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855", + "verification_retry_at": null, + "verified_at": null, + "verification_failure": null, + "verification_retry_count": null, + "verification_checksum": null, + "verification_state": 0, + "verification_started_at": null, + "new_file_path": null +} +``` + ### Publishing a package with the same name or version When you publish a package with the same name and version as an existing package, the new package @@ -76,7 +108,7 @@ API or the UI. #### Do not allow duplicate Generic packages -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293755) in GitLab Free 13.12. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/293755) in GitLab 13.12. To prevent users from publishing duplicate generic packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings) or the UI. diff --git a/doc/user/packages/go_proxy/index.md b/doc/user/packages/go_proxy/index.md index f17910cd46d..1cf3132489a 100644 --- a/doc/user/packages/go_proxy/index.md +++ b/doc/user/packages/go_proxy/index.md @@ -6,11 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Go proxy for GitLab **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27376) in GitLab Premium 13.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/27376) in GitLab 13.1. > - It's deployed behind a feature flag, disabled by default. > - It's disabled for GitLab.com. > - To use it in GitLab self-managed instances, ask a GitLab administrator to [enable it](#enable-the-go-proxy). -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. WARNING: The Go package registry for GitLab is under development and isn't ready for production use due to diff --git a/doc/user/packages/index.md b/doc/user/packages/index.md index a8f1b1998ae..86cfa8986bb 100644 --- a/doc/user/packages/index.md +++ b/doc/user/packages/index.md @@ -10,56 +10,6 @@ The GitLab [Package Registry](package_registry/index.md) acts as a private or pu for a variety of common package managers. You can publish and share packages, which can be easily consumed as a dependency in downstream projects. -WARNING: -Not all package manager formats are ready for production use. To view each format's status, see the -table's **Status** column. - -The Package Registry supports the following formats: - -| Package type | GitLab version | Status | -| ------------ | -------------- |------- | -| [Maven](maven_repository/index.md) | 11.3+ | Stable | -| [npm](npm_registry/index.md) | 11.7+ | Stable | -| [NuGet](nuget_repository/index.md) | 12.8+ | Stable | -| [PyPI](pypi_repository/index.md) | 12.10+ | Stable | -| [Generic packages](generic_packages/index.md) | 13.5+ | Stable | -| [Composer](composer_repository/index.md) | 13.2+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6817) | -| [Conan](conan_repository/index.md) | 12.6+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6816) | -| [Helm](helm_repository/index.md) | 14.1+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6366) | -| [Debian](debian_repository/index.md) | 14.2+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/6057) | -| [Go](go_proxy/index.md) | 13.1+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3043) | -| [Ruby gems](rubygems_registry/index.md) | 13.10+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3200) | - -Status: - -- Alpha: behind a feature flag and not officially supported. -- Beta: several known issues that may prevent expected use. -- Stable: ready for production use. - -You can also use the [API](../../api/packages.md) to administer the Package Registry. - -## Accepting contributions - -The below table lists formats that are not supported, but are accepting Community contributions for. Consider contributing to GitLab. This [development documentation](../../development/packages.md) -guides you through the process. - -<!-- vale gitlab.Spelling = NO --> - -| Format | Status | -| ------ | ------ | -| Chef | [#36889](https://gitlab.com/gitlab-org/gitlab/-/issues/36889) | -| CocoaPods | [#36890](https://gitlab.com/gitlab-org/gitlab/-/issues/36890) | -| Conda | [#36891](https://gitlab.com/gitlab-org/gitlab/-/issues/36891) | -| CRAN | [#36892](https://gitlab.com/gitlab-org/gitlab/-/issues/36892) | -| Opkg | [#36894](https://gitlab.com/gitlab-org/gitlab/-/issues/36894) | -| P2 | [#36895](https://gitlab.com/gitlab-org/gitlab/-/issues/36895) | -| Puppet | [#36897](https://gitlab.com/gitlab-org/gitlab/-/issues/36897) | -| RPM | [#5932](https://gitlab.com/groups/gitlab-org/-/epics/5128) | -| SBT | [#36898](https://gitlab.com/gitlab-org/gitlab/-/issues/36898) | -| Swift | [#12233](https://gitlab.com/gitlab-org/gitlab/-/issues/12233) | -| Vagrant | [#36899](https://gitlab.com/gitlab-org/gitlab/-/issues/36899) | - -<!-- vale gitlab.Spelling = YES --> ## Container Registry The GitLab [Container Registry](container_registry/index.md) is a secure and private registry for container images. It's built on open source software and completely integrated within GitLab. Use GitLab CI/CD to create and publish images. Use the GitLab [API](../../api/container_registry.md) to manage the registry across groups and projects. diff --git a/doc/user/packages/infrastructure_registry/index.md b/doc/user/packages/infrastructure_registry/index.md index 86b85d0c9f5..47f563fd7e7 100644 --- a/doc/user/packages/infrastructure_registry/index.md +++ b/doc/user/packages/infrastructure_registry/index.md @@ -82,7 +82,7 @@ The Infrastructure Registry is automatically enabled. For self-managed instances, a GitLab administrator can [disable](../../../administration/packages/index.md) **Packages & Registries**, -which removes this menu item from the sidebar. **(FREE SELF)** +which removes this menu item from the sidebar. You can also remove the Infrastructure Registry for a specific project: diff --git a/doc/user/packages/maven_repository/index.md b/doc/user/packages/maven_repository/index.md index 17571047353..1ca4bb2759d 100644 --- a/doc/user/packages/maven_repository/index.md +++ b/doc/user/packages/maven_repository/index.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Maven packages in the Package Repository **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5811) in GitLab Premium 11.3. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5811) in GitLab 11.3. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. Publish [Maven](https://maven.apache.org) artifacts in your project's Package Registry. Then, install the packages whenever you need to use them as a dependency. @@ -222,8 +222,8 @@ The `name` must be `Private-Token`. ### Authenticate with a deploy token in Maven -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213566) deploy token authentication in [GitLab Premium](https://about.gitlab.com/pricing/) 13.0. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/213566) deploy token authentication in GitLab 13.0. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. To use a deploy token, add this section to your [`settings.xml`](https://maven.apache.org/settings.html) file. @@ -418,8 +418,8 @@ repositories { ### Group-level Maven endpoint -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8798) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.7. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8798) in GitLab 11.7. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. If you rely on many packages, it might be inefficient to include the `repository` section with a unique URL for each package. Instead, you can use the group-level endpoint for @@ -476,8 +476,8 @@ repositories { ### Instance-level Maven endpoint -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8274) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.7. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/8274) in GitLab 11.7. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. If you rely on many packages, it might be inefficient to include the `repository` section with a unique URL for each package. Instead, you can use the instance-level endpoint for @@ -619,7 +619,7 @@ To delete these older package versions, consider using the Packages API or the U #### Do not allow duplicate Maven packages -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/296895) in GitLab Free 13.9. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/296895) in GitLab 13.9. To prevent users from publishing duplicate Maven packages, you can use the [GraphQl API](../../../api/graphql/reference/index.md#packagesettings) or the UI. diff --git a/doc/user/packages/npm_registry/index.md b/doc/user/packages/npm_registry/index.md index b07ae72e273..03209da7ac8 100644 --- a/doc/user/packages/npm_registry/index.md +++ b/doc/user/packages/npm_registry/index.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # npm packages in the Package Registry **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5934) in GitLab Premium 11.7. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/5934) in GitLab 11.7. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. Publish npm packages in your project's Package Registry. Then install the packages whenever you need to use them as a dependency. @@ -164,8 +164,8 @@ If you encounter an error with [Yarn](https://classic.yarnpkg.com/en/), view ### Authenticate with a CI job token -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9104) in GitLab Premium 12.5. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9104) in GitLab 12.5. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. If you're using npm with GitLab CI/CD, a CI job token can be used instead of a personal access token or deploy token. The token inherits the permissions of the user that generates the pipeline. @@ -208,6 +208,15 @@ Then, you can run `npm publish` either locally or by using GitLab CI/CD. - **GitLab CI/CD:** Set an `NPM_TOKEN` [CI/CD variable](../../../ci/variables/index.md) under your project's **Settings > CI/CD > Variables**. +## Working with private registries + +When working with private repositories, you may want to configure additional settings to ensure a secure communication channel: + +```shell +# Force npm to always require authentication when accessing the registry, even for GET requests. +npm config set always-auth true +``` + ## Package naming convention When you use the [instance-level endpoint](#use-the-gitlab-endpoint-for-npm-packages), only the packages with names in the format of `@scope/package-name` are available. @@ -363,6 +372,10 @@ This rule has a different impact depending on the package name: This aligns with npmjs.org's behavior. However, npmjs.org does not ever let you publish the same version more than once, even if it has been deleted. +## `package.json` limitations + +You can't publish a package if its `package.json` file exceeds 20,000 characters. + ## Install a package npm packages are commonly-installed by using the `npm` or `yarn` commands @@ -427,27 +440,34 @@ and use your organization's URL. The name is case-sensitive and must match the n //gitlab.example.com/api/v4/projects/<your_project_id>/packages/npm/:_authToken= "<your_token>" ``` -### npm dependencies metadata - -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11867) in GitLab Premium 12.6. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. - -In GitLab 12.6 and later, packages published to the Package Registry expose the following attributes to the npm client: - -- name -- version -- dist-tags -- dependencies - - dependencies - - devDependencies - - bundleDependencies - - peerDependencies - - deprecated +### npm metadata + +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/11867) in GitLab 12.6. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. +> - [Improved](https://gitlab.com/gitlab-org/gitlab/-/issues/330929) in GitLab 14.5. + +The GitLab Package Registry exposes the following attributes to the npm client. +These are similar to the [abbreviated metadata format](https://github.com/npm/registry/blob/9e368cf6aaca608da5b2c378c0d53f475298b916/docs/responses/package-metadata.md#abbreviated-metadata-format): + +- `name` +- `versions` + - `name` + - `version` + - `deprecated` + - `dependencies` + - `devDependencies` + - `bundleDependencies` + - `peerDependencies` + - `bin` + - `directories` + - `dist` + - `engines` + - `_hasShrinkwrap` ## Add npm distribution tags -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9425) in GitLab Premium 12.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/9425) in GitLab 12.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. You can add [distribution tags](https://docs.npmjs.com/cli/dist-tag/) to newly-published packages. Tags are optional and can be assigned to only one package at a time. @@ -549,6 +569,8 @@ NPM_TOKEN=<your_token> npm install If you get this error, ensure that: +- The Package Registry is enabled in your project settings. Although the Package Registry is enabled + by default, it's possible to [disable it](../package_registry/#disable-the-package-registry). - Your token is not expired and has appropriate permissions. - A package with the same name or version doesn't already exist within the given scope. - Your NPM package name does not contain a dot `.`. This is a [known issue](https://gitlab.com/gitlab-org/gitlab-ee/issues/10248) @@ -577,6 +599,10 @@ root namespace and therefore cannot be published again using the same name. This is also true even if the prior published package shares the same name, but not the version. +#### Package JSON file is too large + +Make sure that your `package.json` file does not [exceed `20,000` characters](#packagejson-limitations). + ### `npm publish` returns `npm ERR! 500 Internal Server Error - PUT` This is a [known issue](https://gitlab.com/gitlab-org/gitlab/-/issues/238950) in GitLab diff --git a/doc/user/packages/nuget_repository/index.md b/doc/user/packages/nuget_repository/index.md index 2af6dc60078..98cccd72425 100644 --- a/doc/user/packages/nuget_repository/index.md +++ b/doc/user/packages/nuget_repository/index.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # NuGet packages in the Package Registry **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20050) in GitLab Premium 12.8. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20050) in GitLab 12.8. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. > - Symbol package support [added](https://gitlab.com/gitlab-org/gitlab/-/issues/262081) in GitLab 14.1. Publish NuGet packages in your project's Package Registry. Then, install the diff --git a/doc/user/packages/package_registry/index.md b/doc/user/packages/package_registry/index.md index cb4e677687e..3204ac07d6a 100644 --- a/doc/user/packages/package_registry/index.md +++ b/doc/user/packages/package_registry/index.md @@ -6,11 +6,11 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Package Registry **(FREE)** -> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. -With the GitLab Package Registry, you can use GitLab as a private or public registry -for a variety of common package managers. You can publish and share -packages, which can be easily consumed as a dependency in downstream projects. +With the GitLab Package Registry, you can use GitLab as a private or public registry for a variety +of [supported package managers](#supported-package-managers). +You can publish and share packages, which can be consumed as a dependency in downstream projects. ## View packages @@ -48,8 +48,8 @@ Learn more about using the GitLab Package Registry with CI/CD: - [Maven](../maven_repository/index.md#create-maven-packages-with-gitlab-cicd) - [npm](../npm_registry/index.md#publish-an-npm-package-by-using-cicd) - [NuGet](../nuget_repository/index.md#publish-a-nuget-package-by-using-cicd) -- [PyPI](../pypi_repository/#authenticate-with-a-ci-job-token) -- [RubyGems](../rubygems_registry/#authenticate-with-a-ci-job-token) +- [PyPI](../pypi_repository/index.md#authenticate-with-a-ci-job-token) +- [RubyGems](../rubygems_registry/index.md#authenticate-with-a-ci-job-token) If you use CI/CD to build a package, extended activity information is displayed when you view the package details: @@ -124,3 +124,57 @@ Learn how to use the GitLab Package Registry to build your own custom package wo to publish all of your packages to one project. - Publish multiple different packages from one [monorepo project](../workflows/working_with_monorepos.md). + +## Supported package managers + +WARNING: +Not all package manager formats are ready for production use. To view each format's status, see the +table's **Status** column. + +The Package Registry supports the following formats: + +| Package type | GitLab version | Status | +| ------------ | -------------- |------- | +| [Maven](../maven_repository/index.md) | 11.3+ | GA | +| [npm](../npm_registry/index.md) | 11.7+ | GA | +| [NuGet](../nuget_repository/index.md) | 12.8+ | GA | +| [PyPI](../pypi_repository/index.md) | 12.10+ | GA | +| [Generic packages](../generic_packages/index.md) | 13.5+ | GA | +| [Composer](../composer_repository/index.md) | 13.2+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6817) | +| [Conan](../conan_repository/index.md) | 12.6+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6816) | +| [Helm](../helm_repository/index.md) | 14.1+ | [Beta](https://gitlab.com/groups/gitlab-org/-/epics/6366) | +| [Debian](../debian_repository/index.md) | 14.2+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/6057) | +| [Go](../go_proxy/index.md) | 13.1+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3043) | +| [Ruby gems](../rubygems_registry/index.md) | 13.10+ | [Alpha](https://gitlab.com/groups/gitlab-org/-/epics/3200) | + +[Status](https://about.gitlab.com/handbook/product/gitlab-the-product/#generally-available-ga): + +- Alpha: behind a feature flag and not officially supported. +- Beta: several known issues that may prevent expected use. +- GA (Generally Available): ready for production use at any scale. + +You can also use the [API](../../../api/packages.md) to administer the Package Registry. + +## Accepting contributions + +This table lists unsupported package manager formats that we are accepting contributions for. +Consider contributing to GitLab. This [development documentation](../../../development/packages.md) +guides you through the process. + +<!-- vale gitlab.Spelling = NO --> + +| Format | Status | +| ------ | ------ | +| Chef | [#36889](https://gitlab.com/gitlab-org/gitlab/-/issues/36889) | +| CocoaPods | [#36890](https://gitlab.com/gitlab-org/gitlab/-/issues/36890) | +| Conda | [#36891](https://gitlab.com/gitlab-org/gitlab/-/issues/36891) | +| CRAN | [#36892](https://gitlab.com/gitlab-org/gitlab/-/issues/36892) | +| Opkg | [#36894](https://gitlab.com/gitlab-org/gitlab/-/issues/36894) | +| P2 | [#36895](https://gitlab.com/gitlab-org/gitlab/-/issues/36895) | +| Puppet | [#36897](https://gitlab.com/gitlab-org/gitlab/-/issues/36897) | +| RPM | [#5932](https://gitlab.com/groups/gitlab-org/-/epics/5128) | +| SBT | [#36898](https://gitlab.com/gitlab-org/gitlab/-/issues/36898) | +| Swift | [#12233](https://gitlab.com/gitlab-org/gitlab/-/issues/12233) | +| Vagrant | [#36899](https://gitlab.com/gitlab-org/gitlab/-/issues/36899) | + +<!-- vale gitlab.Spelling = YES --> diff --git a/doc/user/packages/pypi_repository/index.md b/doc/user/packages/pypi_repository/index.md index 2d54cfc5f7d..4151346ec98 100644 --- a/doc/user/packages/pypi_repository/index.md +++ b/doc/user/packages/pypi_repository/index.md @@ -6,8 +6,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w # PyPI packages in the Package Registry **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208747) in GitLab Premium 12.10. -> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) to GitLab Free in 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/208747) in GitLab 12.10. +> - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/221259) from GitLab Premium to GitLab Free in 13.3. Publish PyPI packages in your project's Package Registry. Then install the packages whenever you need to use them as a dependency. diff --git a/doc/user/packages/rubygems_registry/index.md b/doc/user/packages/rubygems_registry/index.md index e31bd8bd0bf..05113d0bc10 100644 --- a/doc/user/packages/rubygems_registry/index.md +++ b/doc/user/packages/rubygems_registry/index.md @@ -6,7 +6,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Ruby gems in the Package Registry **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/803) in [GitLab Free](https://about.gitlab.com/pricing/) 13.10. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/803) in GitLab 13.10. WARNING: The Ruby gems package registry for GitLab is under development and isn't ready for production use due to diff --git a/doc/user/packages/workflows/working_with_monorepos.md b/doc/user/packages/workflows/working_with_monorepos.md index 4e431b036de..ab10e746302 100644 --- a/doc/user/packages/workflows/working_with_monorepos.md +++ b/doc/user/packages/workflows/working_with_monorepos.md @@ -4,7 +4,7 @@ group: Package info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Monorepo package management workflows +# Monorepo package management workflows **(FREE)** One project or Git repository can contain multiple different subprojects or submodules that are all packaged and published individually. diff --git a/doc/user/permissions.md b/doc/user/permissions.md index 10147e7f69c..eb79d5099eb 100644 --- a/doc/user/permissions.md +++ b/doc/user/permissions.md @@ -33,8 +33,13 @@ usernames. A GitLab administrator can configure the GitLab instance to ## Project members permissions -The Owner role is only available at the group or personal namespace level (and for instance administrators) and is inherited by its projects. -While Maintainer is the highest project-level role, some actions can only be performed by a personal namespace or group owner, or an instance administrator, who receives all permissions. +A user's role determines what permissions they have on a project. The Owner role provides all permissions but is +available only: + +- For group owners. The role is inherited for a group's projects. +- For Administrators. + +Personal namespace owners have the same permissions as an Owner, but are displayed with the Maintainer role on projects created in their personal namespace. For more information, see [projects members documentation](project/members/index.md). The following table lists project permissions available for each role: @@ -70,7 +75,7 @@ The following table lists project permissions available for each role: | [CI/CD](../ci/index.md):<br>Manage job triggers | | | | ✓ | ✓ | | [CI/CD](../ci/index.md):<br>Manage runners | | | | ✓ | ✓ | | [CI/CD](../ci/index.md):<br>Run Web IDE's Interactive Web Terminals **(ULTIMATE ONLY)** | | | | ✓ | ✓ | -| [CI/CD](../ci/index.md):<br>Use [environment terminals](../ci/environments/index.md#web-terminals) | | | | ✓ | ✓ | +| [CI/CD](../ci/index.md):<br>Use [environment terminals](../ci/environments/index.md#web-terminals-deprecated) | | | | ✓ | ✓ | | [CI/CD](../ci/index.md):<br>Delete pipelines | | | | | ✓ | | [Clusters](infrastructure/clusters/index.md):<br>View [pod logs](project/clusters/kubernetes_pod_logs.md) | | | ✓ | ✓ | ✓ | | [Clusters](infrastructure/clusters/index.md):<br>Manage clusters | | | | ✓ | ✓ | @@ -81,6 +86,15 @@ The following table lists project permissions available for each role: | [GitLab Pages](project/pages/index.md):<br>Manage | | | | ✓ | ✓ | | [GitLab Pages](project/pages/index.md):<br>Manage GitLab Pages domains and certificates | | | | ✓ | ✓ | | [GitLab Pages](project/pages/index.md):<br>Remove GitLab Pages | | | | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [alerts](../operations/incident_management/alerts.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Assign an alert | ✓| ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [incident](../operations/incident_management/incidents.md) | ✓| ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Create [incident](../operations/incident_management/incidents.md) | (*17*) | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [on-call schedules](../operations/incident_management/oncall_schedules.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Participate in on-call rotation | ✓| ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>View [escalation policies](../operations/incident_management/escalation_policies.md) | | ✓ | ✓ | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Manage [on-call schedules](../operations/incident_management/oncall_schedules.md) | | | | ✓ | ✓ | +| [Incident Management](../operations/incident_management/index.md):<br>Manage [escalation policies](../operations/incident_management/escalation_policies.md)| | | | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Add Labels | ✓ (*16*) | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Assign | ✓ (*16*) | ✓ | ✓ | ✓ | ✓ | | [Issues](project/issues/index.md):<br>Create | ✓ | ✓ | ✓ | ✓ | ✓ | @@ -140,7 +154,7 @@ The following table lists project permissions available for each role: | [Projects](project/index.md):<br>Add new team members | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Change [project features visibility](../public_access/public_access.md) level | | | | ✓ (14) | ✓ | | [Projects](project/index.md):<br>Configure [webhooks](project/integrations/webhooks.md) | | | | ✓ | ✓ | -| [Projects](project/index.md):<br>Delete [wiki](project/wiki/index.md) pages | | | | ✓ | ✓ | +| [Projects](project/index.md):<br>Delete [wiki](project/wiki/index.md) pages | | | ✓ | ✓ | ✓ | | [Projects](project/index.md):<br>Edit comments (posted by any user) | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Edit project badges | | | | ✓ | ✓ | | [Projects](project/index.md):<br>Edit project settings | | | | ✓ | ✓ | @@ -169,7 +183,7 @@ The following table lists project permissions available for each role: | [Repository](project/repository/index.md):<br>Enable or disable branch protection | | | | ✓ | ✓ | | [Repository](project/repository/index.md):<br>Enable or disable tag protection | | | | ✓ | ✓ | | [Repository](project/repository/index.md):<br>Manage [push rules](../push_rules/push_rules.md) | | | | ✓ | ✓ | -| [Repository](project/repository/index.md):<br>Push to protected branches | | | | ✓ | ✓ | +| [Repository](project/repository/index.md):<br>Push to protected branches (*5*) | | | | ✓ | ✓ | | [Repository](project/repository/index.md):<br>Turn on or off protected branch push for developers | | | | ✓ | ✓ | | [Repository](project/repository/index.md):<br>Remove fork relationship | | | | | ✓ | | [Repository](project/repository/index.md):<br>Force push to protected branches (*4*) | | | | | | @@ -202,7 +216,7 @@ The following table lists project permissions available for each role: 1. If **Public pipelines** is enabled in **Project Settings > CI/CD**. 1. Not allowed for Guest, Reporter, Developer, Maintainer, or Owner. See [protected branches](project/protected_branches.md). 1. If the [branch is protected](project/protected_branches.md), this depends on the access Developers and Maintainers are given. -1. Guest users can access GitLab [**Releases**](project/releases/index.md) for downloading assets but are not allowed to download the source code nor see repository information like tags and commits. +1. Guest users can access GitLab [**Releases**](project/releases/index.md) for downloading assets but are not allowed to download the source code nor see [repository information like commits and release evidence](project/releases/index.md#view-a-release-and-download-assets). 1. Actions are limited only to records owned (referenced) by user. 1. When [Share Group Lock](group/index.md#prevent-a-project-from-being-shared-with-groups) is enabled the project can't be shared with other groups. It does not affect group with group sharing. 1. For information on eligible approvers for merge requests, see @@ -216,7 +230,9 @@ The following table lists project permissions available for each role: [project visibility](../public_access/public_access.md) is set to private. 1. Attached design files are moved together with the issue even if the user doesn't have the Developer role. -1. Guest users can set metadata (for example, labels, assignees, or milestones) when creating an issue. +1. Guest users can only set metadata (for example, labels, assignees, or milestones) + when creating an issue. They cannot change the metadata on existing issues. +1. In GitLab 14.5 or later, Guests are not allowed to [create incidents](../operations/incident_management/incidents.md#incident-creation). ## Project features permissions @@ -305,7 +321,7 @@ The following table lists group permissions available for each role: | Use security dashboard **(ULTIMATE)** | | | ✓ | ✓ | ✓ | | View group Audit Events | | | ✓ (7) | ✓ (7) | ✓ | | Create subgroup | | | | ✓ (1) | ✓ | -| Delete group wiki pages **(PREMIUM)** | | | | ✓ | ✓ | +| Delete group wiki pages **(PREMIUM)** | | | ✓ | ✓ | ✓ | | Edit epic comments (posted by any user) **(ULTIMATE)** | | | | ✓ (2) | ✓ (2) | | List group deploy tokens | | | | ✓ | ✓ | | Manage [group push rules](group/index.md#group-push-rules) **(PREMIUM)** | | | | ✓ | ✓ | @@ -384,8 +400,10 @@ An administrator can flag a user as external by either of the following methods: 1. On the left sidebar, select **Overview > Users** to create a new user or edit an existing one. There, you can find the option to flag the user as external. -Additionally users can be set as external users using [SAML groups](../integration/saml.md#external-groups) -and [LDAP groups](../administration/auth/ldap/index.md#external-groups). +Additionally users can be set as external users using: + +- [SAML groups](../integration/saml.md#external-groups). +- [LDAP groups](../administration/auth/ldap/ldap_synchronization.md#external-groups). ### Setting new users to external @@ -416,7 +434,7 @@ Be aware that this regex could lead to a ## Free Guest users **(ULTIMATE)** -When a user is given Guest permissions on a project, group, or both, and holds no +When a user is given the Guest role on a project, group, or both, and holds no higher permission level on any other project or group on the GitLab instance, the user is considered a guest user by GitLab and does not consume a license seat. There is no other specific "guest" designation for newly created users. @@ -466,22 +484,20 @@ subscriptions. Project features like wiki and issues can be hidden from users depending on which visibility level you select on project settings. -- Disabled: disabled for everyone -- Only team members: only team members will see even if your project is public or internal -- Everyone with access: everyone can see depending on your project visibility level -- Everyone: enabled for everyone (only available for GitLab Pages) +- Disabled: disabled for everyone. +- Only team members: only team members can see, even if your project is public or internal. +- Everyone with access: everyone can see depending on your project visibility level. +- Everyone: enabled for everyone (only available for GitLab Pages). ## GitLab CI/CD permissions -GitLab CI/CD permissions rely on the role the user has in GitLab. There are four -roles: +GitLab CI/CD permissions rely on the role the user has in GitLab: -- Administrator - Maintainer - Developer - Guest/Reporter -The Administrator role can perform any action on GitLab CI/CD in scope of the GitLab +GitLab administrators can perform any action on GitLab CI/CD in scope of the GitLab instance and project. | Action | Guest, Reporter | Developer |Maintainer| Administrator | diff --git a/doc/user/profile/account/two_factor_authentication.md b/doc/user/profile/account/two_factor_authentication.md index 6fe4b457fac..e4e7e7b9c1a 100644 --- a/doc/user/profile/account/two_factor_authentication.md +++ b/doc/user/profile/account/two_factor_authentication.md @@ -61,7 +61,7 @@ To enable 2FA: - [Authenticator](https://mattrubin.me/authenticator/) - [andOTP](https://github.com/andOTP/andOTP) - [Google Authenticator](https://support.google.com/accounts/answer/1066447?hl=en) - - [Microsoft Authenticator](https://www.microsoft.com/en-us/account/authenticator) + - [Microsoft Authenticator](https://www.microsoft.com/en-us/security/mobile-authenticator-app) - [SailOTP](https://openrepos.net/content/seiichiro0185/sailotp) 1. In the application, add a new entry in one of two ways: - Scan the code presented in GitLab with your device's camera to add the @@ -238,6 +238,9 @@ GitLab officially only supports [YubiKey](https://www.yubico.com/products/) U2F devices, but users have successfully used [SoloKeys](https://solokeys.com/) or [Google Titan Security Key](https://cloud.google.com/titan-security-key). +NOTE: +2FA must be configured before U2F. + The U2F workflow is [supported by](https://caniuse.com/#search=U2F) the following desktop browsers: diff --git a/doc/user/profile/index.md b/doc/user/profile/index.md index 7f16c4e244e..d9f10b58c3f 100644 --- a/doc/user/profile/index.md +++ b/doc/user/profile/index.md @@ -49,7 +49,7 @@ Prerequisites: - Your namespace cannot contain a project with [Container Registry](../packages/container_registry/index.md) tags. - Your namespace cannot have a project that hosts [GitLab Pages](../project/pages/index.md). For more information, - see [this procedure in the GitLab Team Handbook](https://about.gitlab.com/handbook/tools-and-tips/#how-to-change-your-username-at-gitlabcom). + see [this procedure in the GitLab Team Handbook](https://about.gitlab.com/handbook/tools-and-tips/#change-your-username-at-gitlabcom). To change your username: @@ -100,6 +100,18 @@ When visiting the public page of a user, you can only see the projects which you If the [public level is restricted](../admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels), user profiles are only visible to signed-in users. +## User profile README + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/232157) in GitLab 14.5. + +You can add a README section to your profile that can include more information and [formatting](../markdown.md) than +your profile's bio. + +To add a README to your profile: + +1. Create a new public project with the same name as your GitLab username. +1. Create a README file inside this project. The file can be any valid [README or index file](../project/repository/index.md#readme-and-index-files). + ## Add external accounts to your user profile page You can add links to certain other external accounts you might have, like Skype and Twitter. @@ -117,7 +129,7 @@ To add links to other accounts: ## Show private contributions on your user profile page -In the user contribution calendar graph and recent activity list, you can see your [contribution actions](../../api/events.md#action-types) to private projects. +In the user contribution calendar graph and recent activity list, you can see your [contribution actions](../index.md#user-contribution-events) to private projects. To show private contributions: @@ -219,6 +231,12 @@ To set the busy status indicator, either: ## Set your time zone +You can set your local time zone to: + +- Display your local time on your profile, and in places where hovering over your name shows information about you. +- Align your contribution calendar with your local time to better reflect when your contributions were made + ([introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/335343) in GitLab 14.5). + To set your time zone: 1. On the top bar, in the top-right corner, select your avatar. diff --git a/doc/user/profile/notifications.md b/doc/user/profile/notifications.md index 6de09f5538f..9faa4b78f8c 100644 --- a/doc/user/profile/notifications.md +++ b/doc/user/profile/notifications.md @@ -24,6 +24,7 @@ You might receive notifications for one of the following reasons: or edit, or someone mentions you. - You've [enabled notifications in an issue, merge request, or epic](#notifications-on-issues-merge-requests-and-epics). - You've configured notifications for the [project](#change-level-of-project-notifications) or [group](#group-notifications). +- You're subscribed to group or project pipeline notifications via the pipeline emails [integration](../project/integrations/overview.md). NOTE: Administrators can block notifications, preventing them from being sent. @@ -353,3 +354,19 @@ For example, an alert notification email can have one of Expanding the list of events included in the `X-GitLab-NotificationReason` header is tracked in [issue 20689](https://gitlab.com/gitlab-org/gitlab/-/issues/20689). + +## Troubleshooting + +### Pull a list of recipients for notifications + +If you want to pull a list of recipients to receive notifications from a project +(mainly used for troubleshooting custom notifications), +in a Rails console, run `sudo gitlab-rails c` and be sure to update the project name: + +```plaintext +project = Project.find_by_full_path '<project_name>' +merge_request = project.merge_requests.find_by(iid: 1) +current_user = User.first +recipients = NotificationRecipients::BuildService.build_recipients(merge_request, current_user, action: "push_to"); recipients.count +recipients.each { |notify| puts notify.user.username } +``` diff --git a/doc/user/profile/personal_access_tokens.md b/doc/user/profile/personal_access_tokens.md index bdd49b00a15..197ba4647b2 100644 --- a/doc/user/profile/personal_access_tokens.md +++ b/doc/user/profile/personal_access_tokens.md @@ -62,6 +62,10 @@ to the URL. For example: https://gitlab.example.com/-/profile/personal_access_tokens?name=Example+Access+token&scopes=api,read_user,read_registry ``` +WARNING: +Personal access tokens must be treated carefully. Read our [token security considerations](../../security/token_overview.md#security-considerations) +for guidance on managing personal access tokens (for example, setting a short expiry and using minimal scopes). + ## Revoke a personal access token At any time, you can revoke a personal access token. diff --git a/doc/user/project/badges.md b/doc/user/project/badges.md index d54edc7e6d3..9ca11d43864 100644 --- a/doc/user/project/badges.md +++ b/doc/user/project/badges.md @@ -56,11 +56,18 @@ To add this badge to a project: ## Group badges -Badges can be added to a group and are visible on every project's -overview page that's under that group. In this case, they cannot be edited or -deleted on the project level. If you need to have individual badges for each -project, consider adding them on the [project level](#project-badges) or use -[placeholders](#placeholders). +By adding a badge to a group, you add and enforce a project-level badge +for all projects in the group. The group badge is visible on the **Overview** +page of any project that belongs to the group. + +NOTE: +While these badges appear as project-level badges in the codebase, they +cannot be edited or deleted at the project level. + +If you need individual badges for each project, either: + +- Add the badge at the [project level](#project-badges). +- Use [placeholders](#placeholders). To add a new badge to a group: diff --git a/doc/user/project/canary_deployments.md b/doc/user/project/canary_deployments.md index 6d1fb0f6755..77cf04cc7eb 100644 --- a/doc/user/project/canary_deployments.md +++ b/doc/user/project/canary_deployments.md @@ -69,10 +69,14 @@ can easily notice them.  -### Advanced traffic control with Canary Ingress +### Advanced traffic control with Canary Ingress (DEPRECATED) > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/215501) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.6. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212320) to Free in GitLab 13.8. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. Canary deployments can be more strategic with [Canary Ingress](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#canary), which is an advanced traffic routing service that controls incoming HTTP diff --git a/doc/user/project/clusters/add_eks_clusters.md b/doc/user/project/clusters/add_eks_clusters.md index 0db0f14b633..e03e5b10236 100644 --- a/doc/user/project/clusters/add_eks_clusters.md +++ b/doc/user/project/clusters/add_eks_clusters.md @@ -4,12 +4,13 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Connect EKS clusters through cluster certificates **(FREE)** +# Connect EKS clusters through cluster certificates (DEPRECATED) **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22392) in GitLab 12.5. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/22392) in GitLab 12.5. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. WARNING: -Use [Infrastrucure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac) +This feature was deprecated in GitLab 14.5. Use [Infrastrucure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac) to create new clusters. Through GitLab, you can create new clusters and add existing clusters hosted on Amazon Elastic @@ -20,18 +21,10 @@ Kubernetes Service (EKS). If you already have an EKS cluster and want to connect it to GitLab, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). -Alternatively, you can [connect them with cluster certificates](add_existing_cluster.md), -although this method is not recommended for [security implications](../../infrastructure/clusters/connect/index.md#security-implications-for-clusters-connected-with-certificates). - ## Create a new EKS cluster To create a new cluster from GitLab, use [Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac). -Alternatively, you can [create new EKS clusters using cluster certificates](#how-to-create-a-new-cluster-on-eks-through-cluster-certificates-deprecated). -Although still available on the GitLab UI, this method was deprecated -in GitLab 14.0 and is scheduled for removal in GitLab 15.0. -It also has [security implications](../../infrastructure/clusters/connect/index.md#security-implications-for-clusters-connected-with-certificates). - ### How to create a new cluster on EKS through cluster certificates (DEPRECATED) > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) in GitLab 14.0. diff --git a/doc/user/project/clusters/add_existing_cluster.md b/doc/user/project/clusters/add_existing_cluster.md index 3347ef9a437..fcf2583d3ab 100644 --- a/doc/user/project/clusters/add_existing_cluster.md +++ b/doc/user/project/clusters/add_existing_cluster.md @@ -4,16 +4,17 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Connect existing clusters through cluster certificates +# Connect existing clusters through cluster certificates **(DEPRECATED)** -If you have an existing Kubernetes cluster, you can add it to a project, group, -or instance and benefit from the integration with GitLab. +> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. WARNING: -The process described on this page uses cluster certificates to connect your cluster -to GitLab. Although this method still works, it is **no longer recommended**. -To connect your cluster to GitLab, we **recommend** using the [GitLab Kubernetes Agent](../../clusters/agent/index.md) -instead. **(PREMIUM)** +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +To connect your cluster to GitLab, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md) +instead. + +If you have an existing Kubernetes cluster, you can add it to a project, group, +or instance and benefit from the integration with GitLab. ## Prerequisites diff --git a/doc/user/project/clusters/add_gke_clusters.md b/doc/user/project/clusters/add_gke_clusters.md index 0d35e89a560..30be319f2df 100644 --- a/doc/user/project/clusters/add_gke_clusters.md +++ b/doc/user/project/clusters/add_gke_clusters.md @@ -4,9 +4,12 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Connect GKE clusters through cluster certificates **(FREE)** +# Connect GKE clusters through cluster certificates (DEPRECATED) **(FREE)** + +> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. Use [Infrastrucure as Code](../../infrastructure/clusters/connect/new_gke_cluster.md) to create a cluster hosted on Google Kubernetes Engine (GKE). @@ -18,21 +21,13 @@ hosted on Google Kubernetes Engine (GKE). If you already have a GKE cluster and want to connect it to GitLab, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). -Alternatively, you can [connect them with cluster certificates](add_existing_cluster.md), -altough this method is not recommended for [security implications](../../infrastructure/clusters/connect/index.md#security-implications-for-clusters-connected-with-certificates). - ## Create a new GKE cluster from GitLab > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25925) in GitLab 12.4, all the GKE clusters provisioned by GitLab are [VPC-native](https://cloud.google.com/kubernetes-engine/docs/how-to/alias-ips). To create a new GKE cluster from GitLab, use [Infrastructure as Code](../../infrastructure/clusters/connect/new_gke_cluster.md). -Alternatively, you can [create new GKE clusters using cluster certificates](#create-a-new-cluster-on-gke-through-cluster-certificates-deprecated). -Although still available in the GitLab UI, this method was deprecated -in GitLab 14.0 and is scheduled for removal in GitLab 15.0. -It also has [security implications](../../infrastructure/clusters/connect/index.md#security-implications-for-clusters-connected-with-certificates). - -## Create a new cluster on GKE through cluster certificates (DEPRECATED) +## Create a new cluster on GKE through cluster certificates > [Deprecated](https://gitlab.com/groups/gitlab-org/-/epics/6049) in GitLab 14.0. diff --git a/doc/user/project/clusters/add_remove_clusters.md b/doc/user/project/clusters/add_remove_clusters.md index 4f2bc5526e0..49708e3b6aa 100644 --- a/doc/user/project/clusters/add_remove_clusters.md +++ b/doc/user/project/clusters/add_remove_clusters.md @@ -9,9 +9,8 @@ info: To determine the technical writer assigned to the Stage/Group associated w > [Deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) in GitLab 14.0. WARNING: -Creating a new cluster through cluster certificates -is deprecated and no longer recommended. To create a new cluster use -[Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac). +This feature was [deprecated](https://gitlab.com/gitlab-org/gitlab/-/issues/327908) in GitLab 14.0. +To create a new cluster use [Infrastructure as Code](../../infrastructure/iac/index.md#create-a-new-cluster-through-iac). NOTE: Every new Google Cloud Platform (GCP) account receives diff --git a/doc/user/project/clusters/cluster_access.md b/doc/user/project/clusters/cluster_access.md index 452f5727620..510aad821cf 100644 --- a/doc/user/project/clusters/cluster_access.md +++ b/doc/user/project/clusters/cluster_access.md @@ -4,9 +4,15 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Cluster access controls (RBAC or ABAC) +# Access controls with cluster certificates (RBAC or ABAC) (DEPRECATED) **(FREE)** -> Restricted service account for deployment was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51716) in GitLab 11.5. +> - Restricted service account for deployment was [introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/51716) in GitLab 11.5. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +To connect your cluster to GitLab, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md) +instead. When creating a cluster in GitLab, you are asked if you would like to create either: diff --git a/doc/user/project/clusters/deploy_to_cluster.md b/doc/user/project/clusters/deploy_to_cluster.md index 54141fe1103..c3a71ec8585 100644 --- a/doc/user/project/clusters/deploy_to_cluster.md +++ b/doc/user/project/clusters/deploy_to_cluster.md @@ -4,13 +4,14 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Deploy to a Kubernetes cluster with cluster certificates +# Deploy to a Kubernetes cluster with cluster certificates (DEPRECATED) **(FREE)** + +> [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. WARNING: -The process described on this page uses cluster certificates to deploy to your cluster -from GitLab. Although this method still works, it is **no longer recommended**. -To deploy to your cluster from GitLab, we **recommend** using the [GitLab Kubernetes Agent](../../clusters/agent/index.md) -instead. **(PREMIUM)** +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +To connect your cluster to GitLab, use the [GitLab Kubernetes Agent](../../clusters/agent/index.md). +To deploy with the Agent, use the [CI/CD Tunnel](../../clusters/agent/ci_cd_tunnel.md). A Kubernetes cluster can be the destination for a deployment job. If @@ -77,7 +78,7 @@ You can customize the deployment namespace in a few ways: - For **non-managed** clusters, the auto-generated namespace is set in the `KUBECONFIG`, but the user is responsible for ensuring its existence. You can fully customize this value using - [`environment:kubernetes:namespace`](../../../ci/environments/index.md#configure-kubernetes-deployments) + [`environment:kubernetes:namespace`](../../../ci/environments/index.md#configure-kubernetes-deployments-deprecated) in `.gitlab-ci.yml`. When you customize the namespace, existing environments remain linked to their current @@ -100,7 +101,7 @@ combined with *one* of the following: > Introduced in GitLab 8.15. -The Kubernetes integration adds [web terminal](../../../ci/environments/index.md#web-terminals) +The Kubernetes integration adds [web terminal](../../../ci/environments/index.md#web-terminals-deprecated) support to your [environments](../../../ci/environments/index.md). This is based on the `exec` functionality found in Docker and Kubernetes, so you get a new shell session in your existing containers. To use this integration, you diff --git a/doc/user/project/clusters/gitlab_managed_clusters.md b/doc/user/project/clusters/gitlab_managed_clusters.md index 77921ec1dff..ad378be2d9a 100644 --- a/doc/user/project/clusters/gitlab_managed_clusters.md +++ b/doc/user/project/clusters/gitlab_managed_clusters.md @@ -4,10 +4,16 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# GitLab-managed clusters +# GitLab-managed clusters (DEPRECATED) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22011) in GitLab 11.5. > - Became [optional](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/26565) in GitLab 11.11. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +To connect your cluster to GitLab, use the [GitLab Kubernetes Agent](../../../user/clusters/agent/index.md). +To manage applications, use the [Cluster Project Management Template](../../../user/clusters/management_project_template.md). You can choose to allow GitLab to manage your cluster for you. If your cluster is managed by GitLab, resources for your projects are automatically created. See diff --git a/doc/user/project/clusters/index.md b/doc/user/project/clusters/index.md index ac59f874244..c16c6446acd 100644 --- a/doc/user/project/clusters/index.md +++ b/doc/user/project/clusters/index.md @@ -4,19 +4,22 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Project-level Kubernetes clusters **(FREE)** +# Project-level Kubernetes clusters (certificate-based) (DEPRECATED) **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/35954) in GitLab 10.1. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/35954) in GitLab 10.1. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. -[Project-level Kubernetes clusters](../../infrastructure/clusters/connect/index.md#cluster-levels) +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) +in GitLab 14.5. To connect clusters to GitLab, use the +[GitLab Kubernetes Agent](../../clusters/agent/index.md). + +[Project-level](../../infrastructure/clusters/connect/index.md#cluster-levels-deprecated) Kubernetes clusters allow you to connect a Kubernetes cluster to a project in GitLab. You can also [connect multiple clusters](multiple_kubernetes_clusters.md) to a single project. -After connecting a cluster to GitLab, you can benefit from the large number of -[GitLab features available for Kubernetes clusters](../../infrastructure/clusters/index.md) to manage and deploy to your cluster. - ## View your project-level clusters To view project-level Kubernetes clusters: diff --git a/doc/user/project/clusters/kubernetes_pod_logs.md b/doc/user/project/clusters/kubernetes_pod_logs.md index eb0e8d0e91c..19166a1ff8c 100644 --- a/doc/user/project/clusters/kubernetes_pod_logs.md +++ b/doc/user/project/clusters/kubernetes_pod_logs.md @@ -4,10 +4,14 @@ group: Monitor info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Kubernetes Logs **(FREE)** +# Kubernetes Logs (DEPRECATED) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/4752) in [GitLab Ultimate](https://about.gitlab.com/pricing/) 11.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/26383) to [GitLab Free](https://about.gitlab.com/pricing/) 12.9. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. GitLab makes it easy to view the logs of running pods in [connected Kubernetes clusters](index.md). By displaying the logs directly in GitLab diff --git a/doc/user/project/clusters/multiple_kubernetes_clusters.md b/doc/user/project/clusters/multiple_kubernetes_clusters.md index e2eae011b8c..540907bf915 100644 --- a/doc/user/project/clusters/multiple_kubernetes_clusters.md +++ b/doc/user/project/clusters/multiple_kubernetes_clusters.md @@ -4,10 +4,16 @@ group: Configure info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Multiple Kubernetes clusters for a single project +# Multiple clusters per project with cluster certificates (DEPRECATED) **(FREE)** > - Introduced in [GitLab Premium](https://about.gitlab.com/pricing/) 10.3 > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/35094) to GitLab Free in 13.2. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +Using multiple Kubernetes clusters for a single project **with cluster +certificates** was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +To connect clusters to GitLab, use the [GitLab Kubernetes Agent](../../../user/clusters/agent/index.md). You can associate more than one Kubernetes cluster to your project. That way you can have different clusters for different environments, diff --git a/doc/user/project/clusters/protect/container_host_security/index.md b/doc/user/project/clusters/protect/container_host_security/index.md index 5e4df6009f0..c005ce64bb5 100644 --- a/doc/user/project/clusters/protect/container_host_security/index.md +++ b/doc/user/project/clusters/protect/container_host_security/index.md @@ -6,6 +6,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Container Host Security **(FREE)** +NOTE: +In GitLab 14.5, using a certificate to connect GitLab to a Kubernetes cluster is [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). +You can continue using Container Host Security, even though it relies on this certificate-based +method. The work to allow all aspects of Container Host Security to function through the [GitLab Kubernetes Agent](../../../../clusters/agent/index.md) +instead of the certificate-based method can be tracked [in this GitLab issue](https://gitlab.com/gitlab-org/gitlab/-/issues/299350). + Container Host Security in GitLab provides Intrusion Detection and Prevention capabilities that can monitor and (optionally) block activity inside the containers themselves. This is done by leveraging an integration with Falco to provide the monitoring capabilities and an integration with Pod diff --git a/doc/user/project/clusters/protect/container_network_security/index.md b/doc/user/project/clusters/protect/container_network_security/index.md index 3daa48e1811..eb15675da19 100644 --- a/doc/user/project/clusters/protect/container_network_security/index.md +++ b/doc/user/project/clusters/protect/container_network_security/index.md @@ -6,6 +6,12 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Container Network Security **(FREE)** +NOTE: +In GitLab 14.5, using a certificate to connect GitLab to a Kubernetes cluster is [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8). +You can continue using Container Network Security, even though it relies on this certificate-based +method. The work to allow all aspects of Container Network Security to function through the [GitLab Kubernetes Agent](../../../../clusters/agent/index.md) +instead of the certificate-based method can be tracked [in this GitLab issue](https://gitlab.com/gitlab-org/gitlab/-/issues/299350) and [this GitLab Epic](https://gitlab.com/groups/gitlab-org/-/epics/7057). + Container Network Security in GitLab provides basic firewall functionality by leveraging Cilium NetworkPolicies to filter traffic going in and out of the cluster as well as traffic between pods inside the cluster. Container Network Security can be used to enforce L3, L4, and L7 policies and diff --git a/doc/user/project/clusters/serverless/aws.md b/doc/user/project/clusters/serverless/aws.md index 6eafb4530d3..06fa18d80c9 100644 --- a/doc/user/project/clusters/serverless/aws.md +++ b/doc/user/project/clusters/serverless/aws.md @@ -434,11 +434,11 @@ To test the application you deployed, please go to the build log and follow the 1. Click on "Show complete raw" on the upper right-hand corner: -  +  1. Look for HelloWorldApi – API Gateway endpoint similar to shown below: -  +  1. Use curl to test the API. For example: diff --git a/doc/user/project/code_owners.md b/doc/user/project/code_owners.md index 7d51fb59793..c138dc64d19 100644 --- a/doc/user/project/code_owners.md +++ b/doc/user/project/code_owners.md @@ -84,6 +84,10 @@ so that their members also become eligible Code Owners. If you do not invite **Subgroup Y** to **Project A**, but make them Code Owners, their approval of the merge request becomes optional. +Inviting **Subgroup Y** to a parent group of **Project A** +[is not supported](https://gitlab.com/gitlab-org/gitlab/-/issues/288851). To set **Subgroup Y** as +Code Owners, add this group directly to the project itself. + ### Add a group as a Code Owner To set a group as a Code Owner: diff --git a/doc/user/project/deploy_boards.md b/doc/user/project/deploy_boards.md index 6e2635b89f0..15490ab0b94 100644 --- a/doc/user/project/deploy_boards.md +++ b/doc/user/project/deploy_boards.md @@ -5,7 +5,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w type: howto, reference --- -# Deploy boards **(FREE)** +# Deploy boards (DEPRECATED) **(FREE)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/1589) in [GitLab Premium](https://about.gitlab.com/pricing/) 9.0. > - [Moved](https://gitlab.com/gitlab-org/gitlab/-/issues/212320) to GitLab Free in 13.8. @@ -15,6 +15,12 @@ type: howto, reference > - In GitLab 13.11 and earlier, environments in folders do not show deploy boards. > This is [fixed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/60525) in > GitLab 13.12. +> - [Deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. + +WARNING: +This feature was [deprecated](https://gitlab.com/groups/gitlab-org/configure/-/epics/8) in GitLab 14.5. +[An epic exists](https://gitlab.com/groups/gitlab-org/-/epics/2493) +to add this functionality to the [Agent](../index.md). GitLab deploy boards offer a consolidated view of the current health and status of each CI [environment](../../ci/environments/index.md) running on [Kubernetes](https://kubernetes.io), displaying the status diff --git a/doc/user/project/deploy_keys/index.md b/doc/user/project/deploy_keys/index.md index 61dccf1cb1b..c5950347ae9 100644 --- a/doc/user/project/deploy_keys/index.md +++ b/doc/user/project/deploy_keys/index.md @@ -81,10 +81,11 @@ help you access a repository, but there are some notables differences between th [Project maintainers and owners](../../permissions.md#project-members-permissions) can add or enable a deploy key for a project repository: -1. Navigate to the project's **Settings > Repository** page. -1. Expand the **Deploy keys** section. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Deploy keys**. 1. Specify a title for the new deploy key and paste your public SSH key. -1. (Optional) Check **Grant write permissions to this key** to allow `read-write` access. Leave it unchecked for `read-only` access. +1. Optional. To allow `read-write` access, select the **Grant write permissions to this key** checkbox. Leave it unchecked for `read-only` access. There are three lists of project deploy keys: @@ -164,9 +165,10 @@ configuration. [Project maintainers and owners](../../permissions.md#project-members-permissions) can remove or disable a deploy key for a project repository: -1. Navigate to the project's **Settings > Repository** page. -1. Expand the **Deploy keys** section. -1. Select the **{remove}** or **{cancel}** button. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Deploy keys**. +1. Select **Disable** (**{cancel}**). NOTE: Any service that relies on a deploy key stops working after that key is removed. diff --git a/doc/user/project/deploy_tokens/index.md b/doc/user/project/deploy_tokens/index.md index 483de3b21bd..c840f6c8698 100644 --- a/doc/user/project/deploy_tokens/index.md +++ b/doc/user/project/deploy_tokens/index.md @@ -29,14 +29,15 @@ You can create as many deploy tokens as you need from the settings of your project. Alternatively, you can also create [group-scoped deploy tokens](#group-deploy-token). 1. Sign in to your GitLab account. -1. Go to the project (or group) you want to create deploy tokens for. -1. Go to **Settings > Repository**. -1. Expand the **Deploy tokens** section. -1. Choose a name, expiry date (optional), and username (optional) for the token. +1. On the top bar, select **Menu > Projects** or **Menu > Groups** to find your project or group. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Deploy tokens**. +1. Choose a name, and optionally, an expiration date and username for the token. 1. Choose the [desired scopes](#limiting-scopes-of-a-deploy-token). 1. Select **Create deploy token**. -1. Save the deploy token somewhere safe. After you leave or refresh - the page, **you can't access it again**. + +Save the deploy token somewhere safe. After you leave or refresh +the page, **you can't access it again**.  @@ -46,8 +47,12 @@ Deploy tokens expire at midnight UTC on the date you define. ## Revoking a deploy token -To revoke a deploy token, under the **Active deploy tokens** area, -select the respective **Revoke** button. +To revoke a deploy token: + +1. On the top bar, select **Menu > Projects** or **Menu > Groups** to find your project or group. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Deploy tokens**. +1. In the **Active Deploy Tokens** section, by the token you want to revoke, select **Revoke**. ## Limiting scopes of a deploy token diff --git a/doc/user/project/file_lock.md b/doc/user/project/file_lock.md index db8c6f24063..10dcbddac17 100644 --- a/doc/user/project/file_lock.md +++ b/doc/user/project/file_lock.md @@ -212,20 +212,21 @@ requests that modify locked files. Unlock the file to allow changes. To lock a file: 1. Open the file or directory in GitLab. -1. Click the **Lock** button, located near the Web IDE button. +1. On the top right, above the file, select **Lock**. +1. On the confirmation dialog box, select **OK**. -  +If you do not have permission to lock the file, the button is not enabled. -An **Unlock** button is displayed if the file is already locked, and -is disabled if you do not have permission to unlock the file. - -If you did not lock the file, hovering your cursor over the button shows -who locked the file. +To view the user who locked the file (if it was not you), hover over the button. ### View and remove existing locks -The **Locked Files**, accessed from **Project > Repository** left menu, lists -all file and directory locks. Locks can be removed by their author, or any user -with the [Maintainer role](../permissions.md) and above. +To view and remove file locks: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Repository > Locked Files**. This list shows all the files locked either through LFS or GitLab UI. + +Locks can be removed by their author, or any user +with at least the [Maintainer role](../permissions.md). diff --git a/doc/user/project/img/file_lock.png b/doc/user/project/img/file_lock.png Binary files differdeleted file mode 100644 index e881442630b..00000000000 --- a/doc/user/project/img/file_lock.png +++ /dev/null diff --git a/doc/user/project/import/bitbucket.md b/doc/user/project/import/bitbucket.md index cda018a0c37..0c50fc77e33 100644 --- a/doc/user/project/import/bitbucket.md +++ b/doc/user/project/import/bitbucket.md @@ -52,20 +52,18 @@ namespace that started the import process. ## Import your Bitbucket repositories -1. Sign in to GitLab and go to your dashboard. -1. Click on **New project**. - -1. Click on the "Bitbucket Cloud" button. - -  - -1. Grant GitLab access to your Bitbucket account +1. Sign in to GitLab. +1. On the top bar, select **New** (**{plus}**). +1. Select **New project/repository**. +1. Select **Import project**. +1. Select **Bitbucket Cloud**. +1. Log in to Bitbucket and grant GitLab access to your Bitbucket account.  -1. Click on the projects that you'd like to import or **Import all projects**. - You can also filter projects by name and select the namespace under which - each project will be imported. +1. Select the projects that you'd like to import or import all projects. + You can filter projects by name and select the namespace + each project will be imported for.  diff --git a/doc/user/project/import/bitbucket_server.md b/doc/user/project/import/bitbucket_server.md index 2715804b37a..81e7d159a06 100644 --- a/doc/user/project/import/bitbucket_server.md +++ b/doc/user/project/import/bitbucket_server.md @@ -78,10 +78,7 @@ the author's: - `slug` - `displayName` -If the user is not found by any of these properties, the search falls back to the author's -`email` address. - -Alternatively, if there is also no email address, the project creator is set as the author. +If the user is not found by any of these properties, the project creator is set as the author. ##### Enable or disable User assignment by username @@ -104,22 +101,27 @@ Feature.disable(:bitbucket_server_user_mapping_by_username) ## Import your Bitbucket repositories -1. Sign in to GitLab and go to your dashboard. -1. Click on **New project**. -1. Click on the "Bitbucket Server" button. If the button is not present, enable the importer in - **Admin > Application Settings > Visibility and access controls > Import sources**. +Prerequisite: -  +- An administrator must have enabled the importer in + **Admin > Application Settings > Visibility and access controls > Import sources**. -1. Enter your Bitbucket Server credentials. +To import your Bitbucket repositories: -  +1. Sign in to GitLab. +1. On the top bar, select **New** (**{plus}**). +1. Select **New project/repository**. +1. Select **Import project**. +1. Select **Bitbucket Server**. +1. Log in to Bitbucket and grant GitLab access to your Bitbucket account. +1. Select the projects that you'd like to import or import all projects. + You can filter projects by name and select the namespace + each project will be imported for. -1. Click on the projects that you'd like to import or **Import all projects**. - You can also filter projects by name and select the namespace under which each project is - imported. +## Automate group and project import **(PREMIUM)** -  +For information on automating user, group, and project import API calls, see +[Automate group and project import](index.md#automate-group-and-project-import). ## Troubleshooting diff --git a/doc/user/project/import/fogbugz.md b/doc/user/project/import/fogbugz.md index 982bc6d90e8..3458c7fe4a7 100644 --- a/doc/user/project/import/fogbugz.md +++ b/doc/user/project/import/fogbugz.md @@ -16,15 +16,16 @@ users. To import your project from FogBugz: -1. From your GitLab dashboard, select **New project**. -1. Select the **FogBugz** button. -  +1. Sign in to GitLab. +1. On the top bar, select **New** (**{plus}**). +1. Select **New project/repository**. +1. Select **Import project**. +1. Select **FogBugz**. 1. Enter your FogBugz URL, email address, and password. -  1. Create a mapping from FogBugz users to GitLab users.  -1. Select **Import** for the projects you want to import. +1. For the projects you want to import, select **Import**.  -1. After the import finishes, click the link to go to the project +1. After the import finishes, select the link to go to the project dashboard. Follow the directions to push your existing repository.  diff --git a/doc/user/project/import/gitea.md b/doc/user/project/import/gitea.md index 3bbc70b4337..db55330f806 100644 --- a/doc/user/project/import/gitea.md +++ b/doc/user/project/import/gitea.md @@ -38,8 +38,6 @@ that started the import process. The importer page is visible when you create a new project. - - Select the **Gitea** link to start the import authorization process.  diff --git a/doc/user/project/import/github.md b/doc/user/project/import/github.md index eff733b0b3d..e1a81ae1bba 100644 --- a/doc/user/project/import/github.md +++ b/doc/user/project/import/github.md @@ -25,6 +25,7 @@ The following aspects of a project are imported: - Pull request "merged by" information (GitLab.com & 13.7+) - Regular issue and pull request comments - [Git Large File Storage (LFS) Objects](../../../topics/git/lfs/index.md) +- Pull request comments replies in discussions ([GitLab.com & 14.5+](https://gitlab.com/gitlab-org/gitlab/-/issues/336596)) References to pull requests and issues are preserved (GitLab.com & 8.7+), and each imported repository maintains visibility level unless that [visibility @@ -67,7 +68,7 @@ For this association to succeed, each GitHub author and assignee in the reposito must meet one of the following conditions prior to the import: - Have previously logged in to a GitLab account using the GitHub icon. -- Have a GitHub account with a [public-facing email address](https://docs.github.com/en/github/setting-up-and-managing-your-github-user-account/managing-email-preferences/setting-your-commit-email-address) +- Have a GitHub account with a [public-facing email address](https://docs.github.com/en/account-and-profile/setting-up-and-managing-your-github-user-account/managing-email-preferences/setting-your-commit-email-address) that matches their GitLab account's email address. NOTE: @@ -240,3 +241,8 @@ To disable the feature, run this command: # Disable Feature.disable(:github_importer_lower_per_page_limit, group) ``` + +## Automate group and project import **(PREMIUM)** + +For information on automating user, group, and project import API calls, see +[Automate group and project import](index.md#automate-group-and-project-import). diff --git a/doc/user/project/import/gitlab_com.md b/doc/user/project/import/gitlab_com.md index f25b29317a7..bcbc6e09f1b 100644 --- a/doc/user/project/import/gitlab_com.md +++ b/doc/user/project/import/gitlab_com.md @@ -26,3 +26,8 @@ for permission to access your projects. After accepting, you are automatically r To import a project, click "Import". The importer imports your repository and issues. Once the importer is done, a new GitLab project is created with your imported data. + +## Automate group and project import **(PREMIUM)** + +For information on automating user, group, and project import API calls, see +[Automate group and project import](index.md#automate-group-and-project-import). diff --git a/doc/user/project/import/img/bitbucket_server_import_credentials.png b/doc/user/project/import/img/bitbucket_server_import_credentials.png Binary files differdeleted file mode 100644 index 25bcc3ab6e6..00000000000 --- a/doc/user/project/import/img/bitbucket_server_import_credentials.png +++ /dev/null diff --git a/doc/user/project/import/img/bitbucket_server_import_select_project_v12_3.png b/doc/user/project/import/img/bitbucket_server_import_select_project_v12_3.png Binary files differdeleted file mode 100644 index 3f94dd83dd6..00000000000 --- a/doc/user/project/import/img/bitbucket_server_import_select_project_v12_3.png +++ /dev/null diff --git a/doc/user/project/import/img/fogbugz_import_login.png b/doc/user/project/import/img/fogbugz_import_login.png Binary files differdeleted file mode 100644 index 6ba4d443f1a..00000000000 --- a/doc/user/project/import/img/fogbugz_import_login.png +++ /dev/null diff --git a/doc/user/project/import/img/fogbugz_import_select_fogbogz.png b/doc/user/project/import/img/fogbugz_import_select_fogbogz.png Binary files differdeleted file mode 100644 index d207646a6f2..00000000000 --- a/doc/user/project/import/img/fogbugz_import_select_fogbogz.png +++ /dev/null diff --git a/doc/user/project/import/img/import_projects_from_new_project_page.png b/doc/user/project/import/img/import_projects_from_new_project_page.png Binary files differdeleted file mode 100644 index 7c32d3555d1..00000000000 --- a/doc/user/project/import/img/import_projects_from_new_project_page.png +++ /dev/null diff --git a/doc/user/project/import/index.md b/doc/user/project/import/index.md index 887eb546148..6e02a9bf5ab 100644 --- a/doc/user/project/import/index.md +++ b/doc/user/project/import/index.md @@ -102,3 +102,18 @@ After an administrator creates an alias for a project, you can use the alias to repository. For example, if an administrator creates the alias `gitlab` for the project `https://gitlab.com/gitlab-org/gitlab`, you can clone the project with `git clone git@gitlab.com:gitlab.git` instead of `git clone git@gitlab.com:gitlab-org/gitlab.git`. + +## Automate group and project import **(PREMIUM)** + +The GitLab Professional Services team uses [Congregate](https://gitlab.com/gitlab-org/professional-services-automation/tools/migration/congregate) +to orchestrate user, group, and project import API calls. With Congregate, you can migrate data to +GitLab from: + +- Other GitLab instances +- GitHub Enterprise +- GitHub.com +- Bitbucket Server +- Bitbucket Data Center + +See the [Quick Start Guide](https://gitlab.com/gitlab-org/professional-services-automation/tools/migration/congregate/-/blob/master/docs/using-congregate.md#quick-start) +to learn how to use this approach for migrating users, groups, and projects at scale. diff --git a/doc/user/project/import/perforce.md b/doc/user/project/import/perforce.md index f3843396b79..aa256e07b30 100644 --- a/doc/user/project/import/perforce.md +++ b/doc/user/project/import/perforce.md @@ -61,3 +61,7 @@ creating small and efficient Git pack files. So it might be a good idea to spend time and CPU to properly repack your repository before sending it for the first time to your GitLab server. See [this StackOverflow question](https://stackoverflow.com/questions/28720151/git-gc-aggressive-vs-git-repack/). + +## Related topics + +- [Mirror with Perforce Helix with Git Fusion](../repository/mirror/bidirectional.md#mirror-with-perforce-helix-with-git-fusion) diff --git a/doc/user/project/import/repo_by_url.md b/doc/user/project/import/repo_by_url.md index e504f3678a7..0b96238006e 100644 --- a/doc/user/project/import/repo_by_url.md +++ b/doc/user/project/import/repo_by_url.md @@ -23,3 +23,8 @@ You can import your existing repositories by providing the Git URL: <!-- vale gitlab.SubstitutionWarning = YES -->  + +## Automate group and project import **(PREMIUM)** + +For information on automating user, group, and project import API calls, see +[Automate group and project import](index.md#automate-group-and-project-import). diff --git a/doc/user/project/index.md b/doc/user/project/index.md index f3173736e9b..78cd2f8fb79 100644 --- a/doc/user/project/index.md +++ b/doc/user/project/index.md @@ -47,7 +47,7 @@ Projects include the following [features](https://about.gitlab.com/features/): strategy and get reviewed by your team. - [Merge Request Approvals](merge_requests/approvals/index.md): Ask for approval before implementing a change. - - [Fix merge conflicts from the UI](merge_requests/resolve_conflicts.md): View Git diffs from the GitLab UI. + - [Fix merge conflicts from the UI](merge_requests/conflicts.md): View Git diffs from the GitLab UI. - [Review Apps](../../ci/review_apps/index.md): By branch, preview the results of the changes proposed in a merge request. - [Labels](labels.md): Organize issues and merge requests by labels. diff --git a/doc/user/project/integrations/custom_issue_tracker.md b/doc/user/project/integrations/custom_issue_tracker.md index eaab1933b79..d155f91e40b 100644 --- a/doc/user/project/integrations/custom_issue_tracker.md +++ b/doc/user/project/integrations/custom_issue_tracker.md @@ -4,31 +4,40 @@ group: Integrations info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments --- -# Custom issue tracker service **(FREE)** +# Custom issue tracker **(FREE)** -Use a custom issue tracker that is not in the integration list. +You can integrate an [external issue tracker](../../../integration/external-issue-tracker.md) +with GitLab. If your preferred issue tracker is not listed in the +[integrations list](../../../integration/external-issue-tracker.md#integration), +you can enable a custom issue tracker. + +After you enable the custom issue tracker, a link to the issue tracker displays +on the left sidebar in your project. + + + +## Enable a custom issue tracker To enable a custom issue tracker in a project: -1. Go to the [Integrations page](overview.md#accessing-integrations). +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **Custom issue tracker**. 1. Select the checkbox under **Enable integration**. 1. Fill in the required fields: - **Project URL**: The URL to view all the issues in the custom issue tracker. - **Issue URL**: The URL to view an issue in the custom issue tracker. The URL must contain `:id`. - GitLab replaces `:id` with the issue number (for example, - `https://customissuetracker.com/project-name/:id`, which becomes `https://customissuetracker.com/project-name/123`). + GitLab replaces `:id` with the issue number (for example, + `https://customissuetracker.com/project-name/:id`, which becomes + `https://customissuetracker.com/project-name/123`). - **New issue URL**: <!-- The line below was originally added in January 2018: https://gitlab.com/gitlab-org/gitlab/-/commit/778b231f3a5dd42ebe195d4719a26bf675093350 --> - **This URL is not used and removal is planned in a future release.** - Enter any URL here. - For more information, see [issue 327503](https://gitlab.com/gitlab-org/gitlab/-/issues/327503). - -1. Select **Save changes** or optionally select **Test settings**. + **This URL is not used and an [issue exists](https://gitlab.com/gitlab-org/gitlab/-/issues/327503) to remove it.** + Enter any URL. -After you configure and enable the custom issue tracker service, a link appears on the GitLab -project pages. This link takes you to the custom issue tracker. +1. Optional. Select **Test settings**. +1. Select **Save changes**. ## Reference issues in a custom issue tracker diff --git a/doc/user/project/integrations/github.md b/doc/user/project/integrations/github.md index 3177aaefb75..47a81594ca9 100644 --- a/doc/user/project/integrations/github.md +++ b/doc/user/project/integrations/github.md @@ -37,20 +37,20 @@ Complete these steps in GitLab: 1. Ensure that the **Active** toggle is enabled. 1. Paste the token you generated on GitHub. 1. Enter the path to your project on GitHub, such as `https://github.com/username/repository`. -1. (Optional) To disable static status check names, clear the **Static status check names** checkbox. +1. (Optional) To disable static status check names, clear the **Enable static status check names** checkbox. 1. Select **Save changes** or optionally select **Test settings**. After configuring the integration, see [Pipelines for external pull requests](../../../ci/ci_cd_for_external_repos/#pipelines-for-external-pull-requests) to configure pipelines to run for open pull requests. -### Static / dynamic status check names +### Static or dynamic status check names > - Introduced in GitLab 11.5: using static status check names as opt-in option. > - [In GitLab 12.4](https://gitlab.com/gitlab-org/gitlab/-/issues/9931), static status check names is default behavior for new projects. This makes it possible to mark these status checks as **Required** on GitHub. -When **Static status check names** is enabled on the integration page, your +When **Enable static status check names** is checked on the integration page, your GitLab instance host name is appended to a status check name. -When disabled, it uses dynamic status check names and appends the branch name. +When unchecked, it uses dynamic status check names and appends the branch name. diff --git a/doc/user/project/integrations/img/custom_issue_tracker_v14_5.png b/doc/user/project/integrations/img/custom_issue_tracker_v14_5.png Binary files differnew file mode 100644 index 00000000000..e316a2acc39 --- /dev/null +++ b/doc/user/project/integrations/img/custom_issue_tracker_v14_5.png diff --git a/doc/user/project/integrations/img/zentao_product_id.png b/doc/user/project/integrations/img/zentao_product_id.png Binary files differnew file mode 100644 index 00000000000..a91b4c3f82d --- /dev/null +++ b/doc/user/project/integrations/img/zentao_product_id.png diff --git a/doc/user/project/integrations/mattermost.md b/doc/user/project/integrations/mattermost.md index 92e5feefb73..119f219499c 100644 --- a/doc/user/project/integrations/mattermost.md +++ b/doc/user/project/integrations/mattermost.md @@ -54,7 +54,7 @@ Then fill in the integration configuration: To change the bot's username, provide a value. - **Notify only broken pipelines**: If you enable the **Pipeline** event, and you want notifications about failed pipelines only. -- **Branches to be notified**: The branches to send notifications for. +- **Branches for which notifications are to be sent**: The branches to send notifications for. - **Labels to be notified**: (Optional) Labels required for the issue or merge request to trigger a notification. Leave blank to notify for all issues and merge requests. - **Labels to be notified behavior**: When you use the **Labels to be notified** filter, diff --git a/doc/user/project/integrations/microsoft_teams.md b/doc/user/project/integrations/microsoft_teams.md index fac26f8e70c..6679bab745b 100644 --- a/doc/user/project/integrations/microsoft_teams.md +++ b/doc/user/project/integrations/microsoft_teams.md @@ -18,8 +18,8 @@ in Microsoft Teams. To integrate the services, you must: To configure Microsoft Teams to listen for notifications from GitLab: -1. In Microsoft Teams, search for "incoming webhook" in the search bar, and select the - **Incoming Webhook** item: +1. In Microsoft Teams, type `incoming webhook` in the search bar, and select + **Incoming Webhook**:  @@ -34,11 +34,12 @@ To configure Microsoft Teams to listen for notifications from GitLab: After you configure Microsoft Teams to receive notifications, you must configure GitLab to send the notifications: -1. Sign in to GitLab as a user with [Administrator](../../permissions.md) and go - to your project's page. -1. Go to **Settings > Integrations** and select **Microsoft Teams Notification**. -1. Select **Active** to enable the integration. -1. Select the checkbox next to each **Trigger** to enable: +1. Sign in to GitLab as an administrator. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. +1. Select **Microsoft Teams notifications**. +1. To enable the integration, select **Active**. +1. In the **Trigger** section, select the checkbox next to each event to enable it: - Push - Issue - Confidential issue @@ -46,15 +47,15 @@ GitLab to send the notifications: - Note - Confidential note - Tag push - - Pipeline - If you enable this trigger, you can also select **Notify only broken pipelines** to be notified only about failed pipelines. + - Pipeline - Wiki page 1. In **Webhook**, paste the URL you copied when you [configured Microsoft Teams](#configure-microsoft-teams). -1. (Optional) If you enabled the pipeline trigger, you can select the +1. Optional. If you enable the pipeline trigger, select the **Notify only broken pipelines** checkbox to push notifications only when pipelines break. 1. Select the branches you want to send notifications for. -1. Click **Save changes**. +1. Select **Save changes**. -## Resources +## Related topics - [Setting up an incoming webhook on Microsoft Teams](https://docs.microsoft.com/en-us/microsoftteams/platform/webhooks-and-connectors/how-to/connectors-using#setting-up-a-custom-incoming-webhook). diff --git a/doc/user/project/integrations/overview.md b/doc/user/project/integrations/overview.md index 2c154467115..819c17c12fd 100644 --- a/doc/user/project/integrations/overview.md +++ b/doc/user/project/integrations/overview.md @@ -62,6 +62,7 @@ Click on the service links to see further configuration instructions and details | [Unify Circuit](unify_circuit.md) | Send notifications about project events to Unify Circuit. | **{dotted-circle}** No | | [Webex Teams](webex_teams.md) | Receive events notifications. | **{dotted-circle}** No | | [YouTrack](youtrack.md) | Use YouTrack as the issue tracker. | **{dotted-circle}** No | +| [ZenTao](zentao.md) | Use ZenTao as the issue tracker. | **{dotted-circle}** No | ## Push hooks limit diff --git a/doc/user/project/integrations/slack.md b/doc/user/project/integrations/slack.md index a38d2157699..d399c7f2901 100644 --- a/doc/user/project/integrations/slack.md +++ b/doc/user/project/integrations/slack.md @@ -45,7 +45,7 @@ to control GitLab from Slack. Slash commands are configured separately. 1. (Optional) In **Username**, enter the username of the Slack bot that sends the notifications. 1. Select the **Notify only broken pipelines** checkbox to notify only on failures. -1. In the **Branches to be notified** dropdown, select which types of branches +1. In the **Branches for which notifications are to be sent** dropdown, select which types of branches to send notifications for. 1. Leave the **Labels to be notified** field blank to get all notifications, or add labels that the issue or merge request must have to trigger a diff --git a/doc/user/project/integrations/unify_circuit.md b/doc/user/project/integrations/unify_circuit.md index 814c64d8140..daab24a8ab9 100644 --- a/doc/user/project/integrations/unify_circuit.md +++ b/doc/user/project/integrations/unify_circuit.md @@ -21,7 +21,7 @@ In GitLab: 1. Select the checkboxes corresponding to the GitLab events you want to receive in Unify Circuit. 1. Paste the **Webhook URL** that you copied from the Unify Circuit configuration step. 1. Select the **Notify only broken pipelines** checkbox to notify only on failures. -1. In the **Branches to be notified** dropdown, select which types of branches to send notifications for. +1. In the **Branches for which notifications are to be sent** dropdown, select which types of branches to send notifications for. 1. Select `Save changes` or optionally select **Test settings**. Your Unify Circuit conversation now starts receiving GitLab event notifications. diff --git a/doc/user/project/integrations/webhook_events.md b/doc/user/project/integrations/webhook_events.md index 9b07e6322bc..ab70a2d43f4 100644 --- a/doc/user/project/integrations/webhook_events.md +++ b/doc/user/project/integrations/webhook_events.md @@ -9,31 +9,50 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can configure a [webhook](webhooks.md) in your project that triggers when an event occurs. The following events are supported. +Event type | Trigger +---------------------------------------------|----------------------------------------------------------------------------- +[Push event](#push-events) | A push is made to the repository. +[Tag event](#tag-events) | Tags are created or deleted in the repository. +[Issue event](#issue-events) | A new issue is created or an existing issue is updated, closed, or reopened. +[Comment event](#comment-events) | A new comment is made on commits, merge requests, issues, and code snippets. +[Merge request event](#merge-request-events) | A merge request is created, updated, merged, or closed, or a commit is added in the source branch. +[Wiki page event](#wiki-page-events) | A wiki page is created, updated, or deleted. +[Pipeline event](#pipeline-events) | A pipeline status changes. +[Job event](#job-events) | A job status changes. +[Deployment event](#deployment-events) | A deployment starts, succeeds, fails, or is canceled. +[Group member event](#group-member-events) | A user is added or removed from a group, or a user's access level or access expiration date changes. +[Subgroup event](#subgroup-events) | A subgroup is created or removed from a group. +[Feature flag event](#feature-flag-events) | A feature flag is turned on or off. +[Release event](#release-events) | A release is created or updated. + +NOTE: +If an author has no public email listed in their +[GitLab profile](https://gitlab.com/-/profile), the `email` attribute in the +webhook payload displays a value of `[REDACTED]`. + ## Push events -Triggered when you push to the repository except when pushing tags. +Push events are triggered when you push to the repository, except when: -NOTE: -When more than 20 commits are pushed at once, the `commits` webhook -attribute only contains the newest 20 for performance reasons. Loading -detailed commit data is expensive. Note that despite only 20 commits being -present in the `commits` attribute, the `total_commits_count` attribute contains the actual total. +- You push tags. +- A single push includes changes for more than three branches by default + (depending on the [`push_event_hooks_limit` setting](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls)). -NOTE: -If a branch creation push event is generated without new commits being introduced, the -`commits` attribute in the payload is empty. +If you push more than 20 commits at once, the `commits` +attribute in the payload contains information about the newest 20 commits only. +Loading detailed commit data is expensive, so this restriction exists for performance reasons. +The `total_commits_count` attribute contains the actual number of commits. -Also, if a single push includes changes for more than three (by default, depending on -[`push_event_hooks_limit` setting](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls)) -branches, this hook isn't executed. +If you create and push a branch without any new commits, the +`commits` attribute in the payload is empty. -**Request header**: +Request header: ```plaintext X-Gitlab-Event: Push Hook ``` -**Payload example:** +Payload example: ```json { @@ -111,20 +130,19 @@ X-Gitlab-Event: Push Hook ## Tag events -Triggered when you create (or delete) tags to the repository. +Tag events are triggered when you create or delete tags in the repository. -NOTE: -If a single push includes changes for more than three (by default, depending on -[`push_event_hooks_limit` setting](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls)) -tags, this hook is not executed. +This hook is not executed if a single push includes changes for more than three +tags by default (depending on the +[`push_event_hooks_limit` setting](../../../api/settings.md#list-of-settings-that-can-be-accessed-via-api-calls)). -**Request header**: +Request header: ```plaintext X-Gitlab-Event: Tag Push Hook ``` -**Payload example:** +Payload example: ```json { @@ -171,22 +189,26 @@ X-Gitlab-Event: Tag Push Hook ## Issue events -Triggered when a new issue is created or an existing issue was updated/closed/reopened. +Issue events are triggered when a new issue is created or +an existing issue is updated, closed, or reopened. -**Request header**: - -```plaintext -X-Gitlab-Event: Issue Hook -``` - -**Available `object_attributes.action`:** +The available values for `object_attributes.action` in the payload are: - `open` - `close` - `reopen` - `update` -**Payload example:** +The `assignee` and `assignee_id` keys are deprecated +and contain the first assignee only. + +Request header: + +```plaintext +X-Gitlab-Event: Issue Hook +``` + +Payload example: ```json { @@ -329,31 +351,31 @@ X-Gitlab-Event: Issue Hook } ``` -NOTE: -`assignee` and `assignee_id` keys are deprecated and now show the first assignee only. - ## Comment events -Triggered when a new comment is made on commits, merge requests, issues, and code snippets. -The note data is stored in `object_attributes` (for example, `note` or `noteable_type`). The -payload also includes information about the target of the comment. For example, -a comment on an issue includes the specific issue information under the `issue` key. -Valid target types: +Comment events are triggered when a new comment is made on commits, +merge requests, issues, and code snippets. + +The note data is stored in `object_attributes` (for example, `note` or `noteable_type`). +The payload includes information about the target of the comment. For example, +a comment on an issue includes specific issue information under the `issue` key. + +The valid target types are: - `commit` - `merge_request` - `issue` - `snippet` -### Comment on commit +### Comment on a commit -**Request header**: +Request header: ```plaintext X-Gitlab-Event: Note Hook ``` -**Payload example:** +Payload example: ```json { @@ -428,15 +450,15 @@ X-Gitlab-Event: Note Hook } ``` -### Comment on merge request +### Comment on a merge request -**Request header**: +Request header: ```plaintext X-Gitlab-Event: Note Hook ``` -**Payload example:** +Payload example: ```json { @@ -558,15 +580,18 @@ X-Gitlab-Event: Note Hook } ``` -### Comment on issue +### Comment on an issue + +- The `assignee_id` field is deprecated and shows the first assignee only. +- The `event_type` is set to `confidential_note` for confidential issues. -**Request header**: +Request header: ```plaintext X-Gitlab-Event: Note Hook ``` -**Payload example:** +Payload example: ```json { @@ -664,21 +689,15 @@ X-Gitlab-Event: Note Hook } ``` -NOTE: -`assignee_id` field is deprecated and now shows the first assignee only. - -NOTE: -`event_type` is set to `confidential_note` for confidential issues. - -### Comment on code snippet +### Comment on a code snippet -**Request header**: +Request header: ```plaintext X-Gitlab-Event: Note Hook ``` -**Payload example:** +Payload example: ```json { @@ -749,15 +768,13 @@ X-Gitlab-Event: Note Hook ## Merge request events -Triggered when a new merge request is created, an existing merge request was updated/merged/closed or a commit is added in the source branch. +Merge request events are triggered when: -**Request header**: +- A new merge request is created. +- An existing merge request is updated, approved, unapproved, merged, or closed. +- A commit is added in the source branch. -```plaintext -X-Gitlab-Event: Merge Request Hook -``` - -**Available `object_attributes.action`:** +The available values for `object_attributes.action` in the payload are: - `open` - `close` @@ -767,7 +784,13 @@ X-Gitlab-Event: Merge Request Hook - `unapproved` - `merge` -**Payload example:** +Request header: + +```plaintext +X-Gitlab-Event: Merge Request Hook +``` + +Payload example: ```json { @@ -921,17 +944,17 @@ X-Gitlab-Event: Merge Request Hook } ``` -## Wiki Page events +## Wiki page events -Triggered when a wiki page is created, updated or deleted. +Wiki page events are triggered when a wiki page is created, updated, or deleted. -**Request Header**: +Request header: ```plaintext X-Gitlab-Event: Wiki Page Hook ``` -**Payload example**: +Payload example: ```json { @@ -981,18 +1004,18 @@ X-Gitlab-Event: Wiki Page Hook ## Pipeline events +Pipeline events are triggered when the status of a pipeline changes. + In [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53159) and later, the pipeline webhook returns only the latest jobs. -Triggered on status change of Pipeline. - -**Request Header**: +Request header: ```plaintext X-Gitlab-Event: Pipeline Hook ``` -**Payload example**: +Payload example: ```json { @@ -1233,15 +1256,17 @@ X-Gitlab-Event: Pipeline Hook ## Job events -Triggered on status change of a job. +Job events are triggered when the status of a job changes. + +The `commit.id` in the payload is the ID of the pipeline, not the ID of the commit. -**Request Header**: +Request header: ```plaintext X-Gitlab-Event: Job Hook ``` -**Payload example**: +Payload example: ```json { @@ -1303,24 +1328,24 @@ X-Gitlab-Event: Job Hook } ``` -Note that `commit.id` is the ID of the pipeline, not the ID of the commit. - ## Deployment events -Triggered when a deployment: +Deployment events are triggered when a deployment: -- Starts ([Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41214) in GitLab 13.5.) +- Starts ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/41214) in GitLab 13.5) - Succeeds - Fails - Is cancelled -**Request Header**: +The `deployable_id` in the payload is the ID of the CI/CD job. + +Request header: ```plaintext X-Gitlab-Event: Deployment Hook ``` -**Payload example**: +Payload example: ```json { @@ -1363,28 +1388,26 @@ X-Gitlab-Event: Deployment Hook } ``` -Note that `deployable_id` is the ID of the CI job. - ## Group member events **(PREMIUM)** > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/260347) in GitLab 13.7. Member events are triggered when: -- A user is added as a group member -- The access level of a user has changed -- The expiration date for user access has been updated -- A user has been removed from the group +- A user is added as a group member. +- The access level of a user changes. +- The expiration date for user access is updated. +- A user is removed from the group. ### Add member to group -**Request Header**: +Request header: ```plaintext X-Gitlab-Event: Member Hook ``` -**Payload example**: +Payload example: ```json { @@ -1406,13 +1429,13 @@ X-Gitlab-Event: Member Hook ### Update member access level or expiration date -**Request Header**: +Request header: ```plaintext X-Gitlab-Event: Member Hook ``` -**Payload example**: +Payload example: ```json { @@ -1434,13 +1457,13 @@ X-Gitlab-Event: Member Hook ### Remove member from group -**Request Header**: +Request header: ```plaintext X-Gitlab-Event: Member Hook ``` -**Payload example**: +Payload example: ```json { @@ -1466,18 +1489,18 @@ X-Gitlab-Event: Member Hook Subgroup events are triggered when: -- A [subgroup is created in a group](#subgroup-created-in-a-group) -- A [subgroup is removed from a group](#subgroup-removed-from-a-group) +- A [subgroup is created in a group](#create-a-subgroup-in-a-group). +- A [subgroup is removed from a group](#remove-a-subgroup-from-a-group). -### Subgroup created in a group +### Create a subgroup in a group -**Request Header**: +Request header: ```plaintext X-Gitlab-Event: Subgroup Hook ``` -**Payload example**: +Payload example: ```json { @@ -1497,15 +1520,17 @@ X-Gitlab-Event: Subgroup Hook } ``` -### Subgroup removed from a group +### Remove a subgroup from a group -**Request Header**: +This webhook is not triggered when a [subgroup is transferred to a new parent group](../../group/index.md#transfer-a-group). + +Request header: ```plaintext X-Gitlab-Event: Subgroup Hook ``` -**Payload example**: +Payload example: ```json { @@ -1525,20 +1550,17 @@ X-Gitlab-Event: Subgroup Hook } ``` -NOTE: -Webhooks for when a [subgroup is removed from a group](#subgroup-removed-from-a-group) are not triggered when a [subgroup is transferred to a new parent group](../../group/index.md#transfer-a-group) +## Feature flag events -## Feature Flag events +Feature flag events are triggered when a feature flag is turned on or off. -Triggered when a feature flag is turned on or off. - -**Request Header**: +Request header: ```plaintext X-Gitlab-Event: Feature Flag Hook ``` -**Payload example**: +Payload example: ```json { @@ -1580,20 +1602,20 @@ X-Gitlab-Event: Feature Flag Hook ## Release events -Triggered when a release is created or updated. +Release events are triggered when a release is created or updated. -**Request Header**: +The available values for `object_attributes.action` in the payload are: + +- `create` +- `update` + +Request header: ```plaintext X-Gitlab-Event: Release Hook ``` -**Available `object_attributes.action`:** - -- `create` -- `update` - -**Payload example**: +Payload example: ```json { diff --git a/doc/user/project/integrations/webhooks.md b/doc/user/project/integrations/webhooks.md index 0891d48c038..e0405955d3d 100644 --- a/doc/user/project/integrations/webhooks.md +++ b/doc/user/project/integrations/webhooks.md @@ -40,8 +40,14 @@ including: ## Group webhooks **(PREMIUM)** -You can configure a webhook for a group to ensure all projects in the group -receive the same webhook settings. +You can configure a group webhook, which is triggered by events +that occur across all projects in the group. + +Group webhooks can also be configured to listen for events that are +specific to a group, including: + +- [Group member events](webhook_events.md#group-member-events) +- [Subgroup events](webhook_events.md#subgroup-events) ## Configure a webhook diff --git a/doc/user/project/integrations/zentao.md b/doc/user/project/integrations/zentao.md new file mode 100644 index 00000000000..67125c3ebbf --- /dev/null +++ b/doc/user/project/integrations/zentao.md @@ -0,0 +1,42 @@ +--- +stage: Ecosystem +group: Integrations +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# ZenTao product integration **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/338178) in GitLab 14.5. + +[ZenTao](https://www.zentao.net/) is a web-based project management platform. + +## Configure ZenTao + +This integration requires a ZenTao API secret key. + +Complete these steps in ZenTao: + +1. Go to your **Admin** page and select **Develop > Application**. +1. Select **Add Application**. +1. Under **Name** and **Code**, enter a name and a code for the new secret key. +1. Under **Account**, select an existing account name. +1. Select **Save**. +1. Copy the generated key to use in GitLab. + +## Configure GitLab + +Complete these steps in GitLab: + +1. Go to your project and select **Settings > Integrations**. +1. Select **ZenTao**. +1. Turn on the **Active** toggle under **Enable Integration**. +1. Provide the ZenTao configuration information: + - **ZenTao Web URL**: The base URL of the ZenTao instance web interface you're linking to this GitLab project (for example, `example.zentao.net`). + - **ZenTao API URL** (optional): The base URL to the ZenTao instance API. Defaults to the Web URL value if not set. + - **ZenTao API token**: Use the key you generated when you [configured ZenTao](#configure-zentao). + - **ZenTao Product ID**: To display issues from a single ZenTao product in a given GitLab project. The Product ID can be found in the ZenTao product page under **Settings > Overview**. + +  + +1. To verify the ZenTao connection is working, select **Test settings**. +1. Select **Save changes**. diff --git a/doc/user/project/issue_board.md b/doc/user/project/issue_board.md index 8a599608f71..4c35f007fc7 100644 --- a/doc/user/project/issue_board.md +++ b/doc/user/project/issue_board.md @@ -197,21 +197,22 @@ card includes: Users with the [Reporter and higher roles](../permissions.md) can use all the functionality of the issue board feature to create or delete lists. They can also drag issues from one list to another. -## How GitLab orders issues in a list - -When visiting a board, issues appear ordered in any list. You're able to change -that order by dragging the issues. The changed order is saved, so that anybody who visits the same -board later sees the reordering, with some exceptions. +## Ordering issues in a list When an issue is created, the system assigns a relative order value that is greater than the maximum value of that issue's project or root group. This means the issue will be at the bottom of any issue list that it appears in. +When you visit a board, issues appear ordered in any list. You're able to change +that order by dragging the issues. The changed order is saved, so that anybody who visits the same +board later sees the reordering, with some exceptions. + Any time you drag and reorder the issue, its relative order value changes accordingly. Then, any time that issue appears in any board, the ordering is done according to the updated relative order value. If a user in your GitLab instance drags issue `A` above issue `B`, the ordering is maintained when these two issues are subsequently -loaded in any board in the same instance. This could be a different project board or a different group +loaded in any board in the same instance. +This could be a different project board or a different group board, for example. This ordering also affects [issue lists](issues/sorting_issue_lists.md). @@ -593,7 +594,7 @@ You can move issues and lists by dragging them. Prerequisites: -- A minimum of [Reporter](../permissions.md#project-members-permissions) access to a project in GitLab. +- You must have at least the Reporter [role](../permissions.md#project-members-permissions) for a project in GitLab. To move an issue, select the issue card and drag it to another position in its current list or into a different list. Learn about possible effects in [Dragging issues between lists](#dragging-issues-between-lists). diff --git a/doc/user/project/issues/associate_zoom_meeting.md b/doc/user/project/issues/associate_zoom_meeting.md index e020bdee737..aba8c45699c 100644 --- a/doc/user/project/issues/associate_zoom_meeting.md +++ b/doc/user/project/issues/associate_zoom_meeting.md @@ -25,7 +25,7 @@ In an issue, leave a comment using the `/zoom` quick action followed by a valid /zoom https://zoom.us/j/123456789 ``` -If the Zoom meeting URL is valid and you have at least [Reporter permissions](../../permissions.md), +If the Zoom meeting URL is valid and you have at least the Reporter [role](../../permissions.md), a system alert notifies you of its successful addition. The issue's description is automatically edited to include the Zoom link, and a button appears right under the issue's title. @@ -44,5 +44,5 @@ Similarly to adding a Zoom meeting, you can remove it with a quick action: /remove_zoom ``` -If you have at least [Reporter permissions](../../permissions.md), +If you have at least the Reporter [role](../../permissions.md), a system alert notifies you that the meeting URL was successfully removed. diff --git a/doc/user/project/issues/confidential_issues.md b/doc/user/project/issues/confidential_issues.md index 136e8ee2ebb..b8a01f7ccd6 100644 --- a/doc/user/project/issues/confidential_issues.md +++ b/doc/user/project/issues/confidential_issues.md @@ -77,14 +77,14 @@ that prevent leaks of private data. There are two kinds of level access for confidential issues. The general rule is that confidential issues are visible only to members of a project with at -least [Reporter access](../../permissions.md#project-members-permissions). However, a guest user can also create +least the Reporter [role](../../permissions.md#project-members-permissions). However, a guest user can also create confidential issues, but can only view the ones that they created themselves. Confidential issues are also hidden in search results for unprivileged users. -For example, here's what a user with the [Maintainer role](../../permissions.md) and Guest access +For example, here's what a user with the [Maintainer role](../../permissions.md) and the Guest role sees in the project's search results respectively. -| Maintainer role | Guest access | +| Maintainer role | Guest role | |:---------------------------------------------------------------------------------------|:---------------------------------------------------------------------------------| |  |  | diff --git a/doc/user/project/issues/due_dates.md b/doc/user/project/issues/due_dates.md index 94a5fdc3f0f..2c20ccdcee0 100644 --- a/doc/user/project/issues/due_dates.md +++ b/doc/user/project/issues/due_dates.md @@ -7,7 +7,7 @@ info: To determine the technical writer assigned to the Stage/Group associated w # Due dates **(FREE)** Due dates can be used in [issues](index.md) to keep track of deadlines and make sure features are -shipped on time. Users need at least [Reporter permissions](../../permissions.md) +shipped on time. Users need at least the Reporter [role](../../permissions.md) to be able to edit the due date. All users with permission to view the issue can view the due date. diff --git a/doc/user/project/issues/index.md b/doc/user/project/issues/index.md index 9842b0820e6..64838b261ce 100644 --- a/doc/user/project/issues/index.md +++ b/doc/user/project/issues/index.md @@ -49,3 +49,4 @@ To learn how the GitLab Strategic Marketing department uses GitLab issues with [ - [Issues API](../../../api/issues.md) - [Configure an external issue tracker](../../../integration/external-issue-tracker.md) - [Parts of an issue](issue_data_and_actions.md) +- [Tasks](../../tasks.md) diff --git a/doc/user/project/issues/managing_issues.md b/doc/user/project/issues/managing_issues.md index a2fa044be6b..6b3d5d6563a 100644 --- a/doc/user/project/issues/managing_issues.md +++ b/doc/user/project/issues/managing_issues.md @@ -380,12 +380,18 @@ You can also use the `/iteration` [quick action](../quick_actions.md#issues-merge-requests-and-epics) in a comment or description field. -## Real-time sidebar **(FREE SELF)** +## Real-time sidebar -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17589) in GitLab 13.3. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/17589) in GitLab 13.3. Disabled by default. +> - [Enabled on GitLab.com](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/3413) in GitLab 13.9. +> - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/17589) in GitLab 14.5. -Assignees in the sidebar are updated in real time. This feature is **disabled by default**. -To enable it, you need to enable [Action Cable in-app mode](https://docs.gitlab.com/omnibus/settings/actioncable.html). +FLAG: +On self-managed GitLab, by default this feature is available. To hide the feature per project or for your entire instance, ask an administrator to +[disable the feature flags](../../../administration/feature_flags.md) named `real_time_issue_sidebar` and `broadcast_issue_updates`. +On GitLab.com, this feature is available. + +Assignees in the sidebar are updated in real time. ## Similar issues diff --git a/doc/user/project/issues/sorting_issue_lists.md b/doc/user/project/issues/sorting_issue_lists.md index aed346fb504..ebfc723280f 100644 --- a/doc/user/project/issues/sorting_issue_lists.md +++ b/doc/user/project/issues/sorting_issue_lists.md @@ -8,34 +8,59 @@ info: To determine the technical writer assigned to the Stage/Group associated w You can sort a list of issues several ways, including by: -- Blocking **(PREMIUM)** -- Created date -- Due date -- Label priority -- Last updated -- Milestone due date -- Popularity -- Priority -- Title ([introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67234) in GitLab 14.3) -- Weight +- [Blocking issues](#sorting-by-blocking-issues) +- [Created date](#sorting-by-created-date) +- [Due date](#sorting-by-due-date) +- [Label priority](#sorting-by-label-priority) +- [Last updated](#sorting-by-last-updated) +- [Manual sorting](#manual-sorting) +- [Milestone due date](#sorting-by-milestone-due-date) +- [Popularity](#sorting-by-popularity) +- [Priority](#sorting-by-priority) +- [Title](#sorting-by-title) +- [Weight](#sorting-by-weight) The available sorting options can change based on the context of the list. -For sorting by issue priority, see [Label Priority](../labels.md#label-priority). -In group and project issue lists, it is also possible to order issues manually, -similar to [issue boards](../issue_board.md#how-gitlab-orders-issues-in-a-list). +## Sorting by blocking issues **(PREMIUM)** -## Sorting by popularity +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34247/) in GitLab 13.7. -When you select sorting by **Popularity**, the issue order changes to sort descending by the -number of upvotes ([awarded](../../award_emojis.md) "thumbs up" emoji) -on each issue. You can use this to identify issues that are in high demand. +When you sort by **Blocking**, the issue list changes to sort descending by the +number of issues each issue is blocking. + +## Sorting by created date + +When you sort by **Created date**, the issue list changes to sort descending by the issue +creation date. Issues created most recently are first. + +## Sorting by due date + +When you sort by **Due date**, the issue list changes to sort ascending by the issue +[due date](issue_data_and_actions.md#due-date). Issues with the earliest due date are first, +and issues without a due date are last. + +## Sorting by label priority + +When you sort by **Label priority**, the issue list changes to sort descending. +Issues with the highest priority label are first, then all other issues. + +Ties are broken arbitrarily. Only the highest prioritized label is checked, +and labels with a lower priority are ignored. +For more information, see [issue 14523](https://gitlab.com/gitlab-org/gitlab/-/issues/14523). + +To learn more about priority labels, read the [Labels](../labels.md#label-priority) documentation. + +## Sorting by last updated + +When you sort by **Last updated**, the issue list changes to sort by the time of a last +update. Issues changed the most recently are first. ## Manual sorting > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/62178) in GitLab 12.2. -When you select **Manual** sorting, you can change +When you sort by **Manual** order, you can change the order by dragging and dropping the issues. The changed order persists, and everyone who visits the same list sees the updated issue order, with some exceptions. @@ -48,13 +73,47 @@ the updated relative order value is used for the ordering. So, if anyone drags issue `A` above issue `B` in your GitLab instance, this ordering is maintained whenever they appear together in any list. -This ordering also affects [issue boards](../issue_board.md#how-gitlab-orders-issues-in-a-list). +This ordering also affects [issue boards](../issue_board.md#ordering-issues-in-a-list). Changing the order in an issue list changes the ordering in an issue board, -and vice versa. +and the other way around. -## Sorting by blocking issues **(PREMIUM)** +## Sorting by milestone due date -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/34247/) in GitLab 13.7. +When you sort by **Milestone due date**, the issue list changes to sort ascending by the +assigned milestone due date. Issues with milestones with the earliest due date are first, +then issues with a milestone without a due date. + +## Sorting by popularity + +When you sort by **Popularity**, the issue order changes to sort descending by the +number of upvotes ([awarded](../../award_emojis.md) a "thumbs up" emoji) +on each issue. You can use this to identify issues that are in high demand. + +## Sorting by priority + +When you sort by **Priority**, the issue order changes to sort in this order: + +1. Issues with milestones that have due dates, where the soonest assigned milestone is listed first. +1. Issues with milestones with no due dates. +1. Issues with a higher priority label. +1. Issues without a prioritized label. + +To learn more about priority, read the [Labels](../labels.md#label-priority) documentation. + +## Sorting by title + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/67234) in GitLab 14.3. + +When you sort by **Title**, the issue order changes to sort alphabetically by the issue +title in this order: + +- Emoji +- Special characters +- Numbers +- Letters: first Latin, then accented (for example, `ö`) + +## Sorting by weight -When you select to sort by **Blocking**, the issue list changes to sort descending by the -number of issues each issue is blocking. You can use this to determine the critical path for your backlog. +When you sort by **Weight**, the issue list changes to sort ascending by the +[issue weight](issue_weight.md). +Issues with lowest weight are first, and issues without a weight are last. diff --git a/doc/user/project/members/index.md b/doc/user/project/members/index.md index f9788ef18ec..adf0a115c6e 100644 --- a/doc/user/project/members/index.md +++ b/doc/user/project/members/index.md @@ -120,8 +120,8 @@ To remove a member from a project: 1. Optional. In the confirmation box, select the **Also unassign this user from related issues and merge requests** checkbox. 1. To prevent leaks of sensitive information from private projects, verify the - user has not forked the private repository. Existing forks continue to receive - changes from the upstream project. You may also want to configure your project + user has not forked the private repository or created webhooks. Existing forks continue to receive + changes from the upstream project, and webhooks continue to receive updates. You may also want to configure your project to prevent projects in a group [from being forked outside their group](../../group/index.md#prevent-project-forking-outside-group). 1. Select **Remove member**. diff --git a/doc/user/project/merge_requests/accessibility_testing.md b/doc/user/project/merge_requests/accessibility_testing.md index 2bc6d5bb148..8f803f9207c 100644 --- a/doc/user/project/merge_requests/accessibility_testing.md +++ b/doc/user/project/merge_requests/accessibility_testing.md @@ -21,6 +21,9 @@ measuring the accessibility of web sites, and has built a simple This job outputs accessibility violations, warnings, and notices for each page analyzed to a file called `accessibility`. +From [GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/73309), the version of `pa11y` uses +[WCAG 2.1 rules](https://www.w3.org/TR/WCAG21/#new-features-in-wcag-2-1), which may report more issues. + ## Accessibility merge request widget > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/39425) in GitLab 13.0 behind the disabled [feature flag](../../../administration/feature_flags.md) `:accessibility_report_view`. diff --git a/doc/user/project/merge_requests/approvals/index.md b/doc/user/project/merge_requests/approvals/index.md index aff55419a88..d873f715557 100644 --- a/doc/user/project/merge_requests/approvals/index.md +++ b/doc/user/project/merge_requests/approvals/index.md @@ -3,7 +3,7 @@ stage: Create group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference, concepts -disqus_identifier: 'https://docs.gitlab.com/ee/user/project/merge_requests/merge_request_approvals.html' +disqus_identifier: 'https://docs.gitlab.com/ee/user/project/merge_requests/approvals/index.html' --- # Merge request approvals **(FREE)** diff --git a/doc/user/project/merge_requests/approvals/rules.md b/doc/user/project/merge_requests/approvals/rules.md index b422982c0e7..1249aa826fa 100644 --- a/doc/user/project/merge_requests/approvals/rules.md +++ b/doc/user/project/merge_requests/approvals/rules.md @@ -167,7 +167,7 @@ for protected branches. **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/40491) in GitLab 13.4. > - Moved to GitLab Premium in 13.9. -You may need to grant users with [Reporter permissions](../../../permissions.md#project-members-permissions), +You may have to grant users with the Reporter [role](../../../permissions.md#project-members-permissions) permission to approve merge requests before they can merge to a protected branch. Some users (like managers) may not need permission to push or merge code, but still need oversight on proposed work. To enable approval permissions for these users without diff --git a/doc/user/project/merge_requests/approvals/settings.md b/doc/user/project/merge_requests/approvals/settings.md index 1c56e91ed6b..56e93741c1a 100644 --- a/doc/user/project/merge_requests/approvals/settings.md +++ b/doc/user/project/merge_requests/approvals/settings.md @@ -39,7 +39,7 @@ By default, the author of a merge request cannot approve it. To change this sett 1. Go to your project and select **Settings > General**. 1. Expand **Merge request (MR) approvals**. -1. Clear the **Prevent MR approval by the author** checkbox. +1. Clear the **Prevent approval by author** checkbox. 1. Select **Save changes**. Authors can edit the approval rule in an individual merge request and override @@ -64,14 +64,20 @@ their own. To do this: 1. Go to your project and select **Settings > General**. 1. Expand **Merge request (MR) approvals**. -1. Select the **Prevent MR approvals from users who make commits to the MR** checkbox. +1. Select the **Prevent approvals by users who add commits** checkbox. If this checkbox is cleared, an administrator has disabled it [at the instance level](../../../admin_area/merge_requests_approvals.md), and it can't be changed at the project level. 1. Select **Save changes**. -Even with this configuration, [code owners](../../code_owners.md) who contribute -to a merge request can approve merge requests that affect files they own. +Depending on your version of GitLab, [code owners](../../code_owners.md) who commit +to a merge request may or may not be able to approve the work: + +- In GitLab 13.10 and earlier, [code owners](../../code_owners.md) who commit + to a merge request can approve it, even if the merge request affects files they own. +- In [GitLab 13.11 and later](https://gitlab.com/gitlab-org/gitlab/-/issues/331548), + [code owners](../../code_owners.md) who commit + to a merge request cannot approve it, when the merge request affects files they own. To learn more about the [differences between authors and committers](https://git-scm.com/book/en/v2/Git-Basics-Viewing-the-Commit-History), read the official Git documentation for an explanation. @@ -84,7 +90,7 @@ on merge requests, you can disable this setting: 1. Go to your project and select **Settings > General**. 1. Expand **Merge request (MR) approvals**. -1. Select the **Prevent users from modifying MR approval rules in merge requests** checkbox. +1. Select the **Prevent editing approval rules in merge requests** checkbox. 1. Select **Save changes**. This change affects all open merge requests. @@ -102,7 +108,7 @@ permission enables an electronic signature for approvals, such as the one define [sign-in restrictions documentation](../../../admin_area/settings/sign_in_restrictions.md#password-authentication-enabled). 1. Go to your project and select **Settings > General**. 1. Expand **Merge request (MR) approvals**. -1. Select the **Require user password for approvals** checkbox. +1. Select the **Require user password to approve** checkbox. 1. Select **Save changes**. ## Remove all approvals when commits are added to the source branch @@ -113,7 +119,7 @@ when more changes are added to it: 1. Go to your project and select **Settings > General**. 1. Expand **Merge request (MR) approvals**. -1. Select the **Require new approvals when new commits are added to an MR** checkbox. +1. Select the **Remove all approvals when commits are added to the source branch** checkbox. 1. Select **Save changes**. Approvals aren't reset when a merge request is [rebased from the UI](../fast_forward_merge.md) @@ -133,21 +139,23 @@ coverage. To learn more, see [Coverage check approval rule](../../../../ci/pipelines/settings.md#coverage-check-approval-rule). -## Merge request approval settings cascading +## Settings cascading -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285410) in GitLab 14.4. [Deployed behind the `group_merge_request_approval_settings_feature_flag` flag](../../../../administration/feature_flags.md), disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/285410) in GitLab 14.4. [Deployed behind the `group_merge_request_approval_settings_feature_flag` flag](../../../../administration/feature_flags.md), disabled by default. +> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/285410) in GitLab 14.5. FLAG: -On self-managed GitLab, by default this feature is not available. To make it available per group, ask an administrator to [enable the `group_merge_request_approval_settings_feature_flag` flag](../../../../administration/feature_flags.md). On GitLab.com, this feature is not available. -You should not use this feature for production environments +On self-managed GitLab, by default this feature is available. To hide the feature per group, ask an administrator to [disable the feature flag](../../../../administration/feature_flags.md) named `group_merge_request_approval_settings_feature_flag`. On GitLab.com, this feature is available. You can also enforce merge request approval settings: -- At the [instance level](../../../admin_area/merge_requests_approvals.md), which apply to all groups on an instance and, therefore, all - projects. -- On a [top-level group](../../../group/index.md#group-approval-rules), which apply to all subgroups and projects. +- At the [instance level](../../../admin_area/merge_requests_approvals.md), which apply to all groups + on an instance and, therefore, all projects. +- On a [top-level group](../../../group/index.md#group-approval-rules), which apply to all subgroups + and projects. -If the settings are inherited by a group or project, they cannot be overridden by the group or project that inherited them. +If the settings are inherited by a group or project, they cannot be changed in the group or project +that inherited them. ## Related links diff --git a/doc/user/project/merge_requests/authorization_for_merge_requests.md b/doc/user/project/merge_requests/authorization_for_merge_requests.md index 339f67f828f..4ae59a76a9a 100644 --- a/doc/user/project/merge_requests/authorization_for_merge_requests.md +++ b/doc/user/project/merge_requests/authorization_for_merge_requests.md @@ -40,7 +40,7 @@ protected branch. ## Forking workflow With the forking workflow, maintainers get the [Maintainer role](../../permissions.md) and regular -developers get Reporter access to the authoritative repository, which prohibits +developers get the Reporter role on the authoritative repository, which prohibits them from pushing any changes to it. Developers create forks of the authoritative project and push their feature diff --git a/doc/user/project/merge_requests/code_quality.md b/doc/user/project/merge_requests/code_quality.md index e9f1874eb96..9bfbbd8fc6f 100644 --- a/doc/user/project/merge_requests/code_quality.md +++ b/doc/user/project/merge_requests/code_quality.md @@ -343,8 +343,7 @@ It's possible to have a custom tool provide Code Quality reports in GitLab. To do this: 1. Define a job in your `.gitlab-ci.yml` file that generates the - [Code Quality report - artifact](../../../ci/yaml/index.md#artifactsreportscodequality). + [Code Quality report artifact](../../../ci/yaml/index.md#artifactsreportscodequality). 1. Configure your tool to generate the Code Quality report artifact as a JSON file that implements a subset of the [Code Climate spec](https://github.com/codeclimate/platform/blob/master/spec/analyzers/SPEC.md#data-types). diff --git a/doc/user/project/merge_requests/commit_templates.md b/doc/user/project/merge_requests/commit_templates.md new file mode 100644 index 00000000000..b615c86288c --- /dev/null +++ b/doc/user/project/merge_requests/commit_templates.md @@ -0,0 +1,51 @@ +--- +stage: Create +group: Code Review +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference, howto +--- + +# Commit message templates **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/20263) in GitLab 14.5. + +## Merge commit message template + +As a project maintainer, you're able to configure merge commit message template. It will be used during merge to +create commit message. Template uses similar syntax to +[review suggestions](reviews/suggestions.md#configure-the-commit-message-for-applied-suggestions). + +Default merge commit message can be recreated using following template: + +```plaintext +Merge branch '%{source_branch}' into '%{target_branch}' + +%{title} + +%{issues} + +See merge request %{reference} +``` + +This commit message can be customized to follow any guidelines you might have. +To do so, expand the **Merge requests** tab within your project's **General** +settings and change the **Merge commit message template** text: + + + +You can use static text and following variables: + +| Variable | Description | Output example | +|--------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------| +| `%{source_branch}` | The name of the branch that is being merged. | `my-feature-branch` | +| `%{target_branch}` | The name of the branch that the changes are applied to. | `master` | +| `%{title}` | Title of the merge request. | Fix stuff | +| `%{issues}` | String with phrase "Closes <issue numbers>" with all issues mentioned in the MR description matching [issue closing patterns](../issues/managing_issues.md#closing-issues-automatically). It will be empty when no issues were mentioned. | `Closes #465, #190 and #400` | +| `%{description}` | Description of the merge request. | Merge request description.<br>Can be multiline. | +| `%{reference}` | Reference to the merge request. | group-name/project-name!72359 | + +NOTE: +Empty variables that are the only word in a line will be removed along with all newline characters preceding it. + +Merge commit template field has a limit of 500 characters. This limit only applies to the template +itself. diff --git a/doc/user/project/merge_requests/conflicts.md b/doc/user/project/merge_requests/conflicts.md new file mode 100644 index 00000000000..dc128f89fcd --- /dev/null +++ b/doc/user/project/merge_requests/conflicts.md @@ -0,0 +1,177 @@ +--- +stage: Create +group: Code Review +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +type: reference, concepts +--- + +# Merge conflicts **(FREE)** + +_Merge conflicts_ happen when the two branches in a merge request (the source and target) each have different +changes, and you must decide which change to accept. In a merge request, Git compares +the two versions of the files line by line. In most cases, GitLab can merge changes +together. However, if two branches both change the same lines, GitLab blocks the merge, +and you must choose which change you want to keep. + +A merge request cannot merge until you either: + +- Create a merge commit. +- Resolve the conflict through a rebase. + + + +## Conflicts you can resolve in the user interface + +If your merge conflict meets all of the following conditions, you can resolve the +merge conflict in the GitLab user interface: + +- The file is text, not binary. +- The file is in a UTF-8 compatible encoding. +- The file does not already contain conflict markers. +- The file, with conflict markers added, is less than 200 KB in size. +- The file exists under the same path in both branches. + +If any file in your merge request contains conflicts, but can't meet all of these +criteria, you must resolve the conflict manually. + +## Conflicts GitLab can't detect + +GitLab does not detect conflicts when both branches rename a file to different names. +For example, these changes don't create a conflict: + +1. On branch `a`, doing `git mv example.txt example1.txt` +1. On branch `b`, doing `git mv example1.txt example3.txt`. + +When these branches merge, both `example1.txt` and `example3.txt` are present. + +## Methods of resolving conflicts + +GitLab shows [conflicts available for resolution](#conflicts-you-can-resolve-in-the-user-interface) +in the user interface, and you can also resolve conflicts locally through the command line: + +- [Interactive mode](#resolve-conflicts-in-interactive-mode): UI method best for + conflicts that only require you to select which version of a line to keep, without edits. +- [Inline editor](#resolve-conflicts-in-the-inline-editor): UI method best for more complex conflicts that require you to + edit lines and manually blend changes together. +- [Command line](#resolve-conflicts-from-the-command-line): provides complete control over the most complex conflicts. + +## Resolve conflicts in interactive mode + +To resolve less-complex conflicts from the GitLab user interface: + +1. Go to your merge request. +1. Select **Overview**, and scroll to the merge request reports section. +1. Find the merge conflicts message, and select **Resolve conflicts**. + GitLab shows a list of files with merge conflicts. The conflicts are + highlighted: + +  +1. For each conflict, select **Use ours** or **Use theirs** to mark the version + of the conflicted lines you want to keep. This decision is known as + "resolving the conflict." +1. Enter a **Commit message**. +1. Select **Commit to source branch**. + +Resolving conflicts merges the target branch of the merge request into the +source branch, using the version of the text you chose. If the source branch is +`feature` and the target branch is `main`, these actions are similar to running +`git checkout feature; git merge main` locally. + +## Resolve conflicts in the inline editor + +Some merge conflicts are more complex, requiring you to manually modify lines to +resolve their conflicts. Use the merge conflict resolution editor to resolve complex +conflicts in the GitLab interface: + +1. Go to your merge request. +1. Select **Overview**, and scroll to the merge request reports section. +1. Find the merge conflicts message, and select **Resolve conflicts**. + GitLab shows a list of files with merge conflicts. +1. Select **Edit inline** to open the editor: +  +1. After you resolve the conflict, enter a **Commit message**. +1. Select **Commit to source branch**. + +## Resolve conflicts from the command line + +While most conflicts can be resolved through the GitLab user interface, some are too complex. +Complex conflicts are best fixed locally, from the command line, to give you the +most control over each change: + +1. Open the terminal and check out your feature branch. For example, `my-feature-branch`: + + ```shell + git checkout my-feature-branch + ``` + +1. [Rebase your branch](../../../topics/git/git_rebase.md#regular-rebase) against the + target branch (here, `main`) so Git prompts you with the conflicts: + + ```shell + git fetch + git rebase origin/main + ``` + +1. Open the conflicting file in your preferred code editor. +1. Find the conflict block: + - It begins with the marker: `<<<<<<< HEAD`. + - Next, it displays your changes. + - The marker `=======` indicates the end of your changes. + - Next, it displays the latest changes in the target branch. + - The marker `>>>>>>>` indicates the end of the conflict. +1. Edit the file: + 1. Choose which version (before or after `=======`) you want to keep. + 1. Delete the version you don't want to keep. + 1. Delete the conflict markers. +1. Save the file. +1. Repeat the process for each file that contains conflicts. +1. Stage your changes in Git: + + ```shell + git add . + ``` + +1. Commit your changes: + + ```shell + git commit -m "Fix merge conflicts" + ``` + +1. Continue the rebase: + + ```shell + git rebase --continue + ``` + + WARNING: + Up to this point, you can run `git rebase --abort` to stop the process. + Git aborts the rebase and rolls back the branch to the state you had before + running `git rebase`. + After you run `git rebase --continue`, you cannot abort the rebase. + +1. [Force-push](../../../topics/git/git_rebase.md#force-push) the changes to your + remote branch. + +## Merge commit strategy + +GitLab resolves conflicts by creating a merge commit in the source branch, but +does not merge it into the target branch. You can then review and test the +merge commit. Verify it contains no unintended changes and doesn't break your build. + +## Related topics + +- [Introduction to Git rebase and force-push](../../../topics/git/git_rebase.md). +- [Git GUI apps](https://git-scm.com/downloads/guis) to help you visualize the + differences between branches and resolve them. + +<!-- ## Troubleshooting + +Include any troubleshooting steps that you can foresee. If you know beforehand what issues +one might have when setting this up, or when something is changed, or on upgrading, it's +important to describe those, too. Think of things that may go wrong and include them here. +This is important to minimize requests for support, and to avoid doc comments with +questions that you know someone might ask. + +Each scenario can be a third-level heading, e.g. `### Getting error message X`. +If you have none to add when creating a doc, leave this section in place +but commented out to help encourage others to add to it in the future. --> diff --git a/doc/user/project/merge_requests/fast_forward_merge.md b/doc/user/project/merge_requests/fast_forward_merge.md index 7edc379399b..078f8048900 100644 --- a/doc/user/project/merge_requests/fast_forward_merge.md +++ b/doc/user/project/merge_requests/fast_forward_merge.md @@ -38,6 +38,8 @@ Now, when you visit the merge request page, you can accept it If a fast-forward merge is not possible but a conflict free rebase is possible, a rebase button is offered. +The rebase action is also available as a [quick action command: `/rebase`](../../../topics/git/git_rebase.md#rebase-from-the-gitlab-ui). +  If the target branch is ahead of the source branch and a conflict free rebase is diff --git a/doc/user/project/merge_requests/getting_started.md b/doc/user/project/merge_requests/getting_started.md index cee4df1f61e..006e6d4a8aa 100644 --- a/doc/user/project/merge_requests/getting_started.md +++ b/doc/user/project/merge_requests/getting_started.md @@ -172,11 +172,6 @@ is set for deletion, the merge request widget displays the > - [Disabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/320902) in GitLab 13.9. > - [Enabled on self-managed](https://gitlab.com/gitlab-org/gitlab/-/issues/320895) GitLab 13.10. -FLAG: -On self-managed GitLab, by default this feature is available. To hide the feature, -ask an administrator to -[disable the `retarget_merge_requests` flag](../../../administration/feature_flags.md). - In specific circumstances, GitLab can retarget the destination branch of open merge request, if the destination branch merges while the merge request is open. Merge requests are often chained in this manner, with one merge request diff --git a/doc/user/project/merge_requests/img/merge_commit_message_template_v14_5.png b/doc/user/project/merge_requests/img/merge_commit_message_template_v14_5.png Binary files differnew file mode 100644 index 00000000000..f18ca640d38 --- /dev/null +++ b/doc/user/project/merge_requests/img/merge_commit_message_template_v14_5.png diff --git a/doc/user/project/merge_requests/img/merge_request_tab_position_v13_11.png b/doc/user/project/merge_requests/img/merge_request_tab_position_v13_11.png Binary files differdeleted file mode 100644 index 52c715840f1..00000000000 --- a/doc/user/project/merge_requests/img/merge_request_tab_position_v13_11.png +++ /dev/null diff --git a/doc/user/project/merge_requests/img/project_merge_requests_list_view_v13_5.png b/doc/user/project/merge_requests/img/project_merge_requests_list_view_v13_5.png Binary files differdeleted file mode 100644 index 625d47b1142..00000000000 --- a/doc/user/project/merge_requests/img/project_merge_requests_list_view_v13_5.png +++ /dev/null diff --git a/doc/user/project/merge_requests/index.md b/doc/user/project/merge_requests/index.md index 2c062c2c592..54b97eb5732 100644 --- a/doc/user/project/merge_requests/index.md +++ b/doc/user/project/merge_requests/index.md @@ -17,75 +17,50 @@ Merge requests include: - A comment section for discussion threads. - The list of commits. -To get started, read the [introduction to merge requests](getting_started.md). +Read more about [how to get started](getting_started.md). -## Merge request tabs +## View merge requests -Merge requests contain tabs at the top of the page to help you navigate to -important parts of the merge request: +You can view merge requests for your project, group, or yourself. - +### View merge requests for a project -- **Overview**: Contains the description, notifications from pipelines, and a - discussion area for [comment threads](../../discussions/index.md#resolve-a-thread) - and [code suggestions](reviews/suggestions.md). The right sidebar provides fields - to add assignees, reviewers, labels, and a milestone to your work, and the - [merge request widgets area](widgets.md) reports results from pipelines and tests. -- **Commits**: Contains a list of commits added to this merge request. For more - information, read [Commits tab in merge requests](commits.md). -- **Pipelines**: If configured, contains a list of recent [GitLab CI/CD](../../../ci/index.md) - pipelines and their status. -- **Changes**: Contains the diffs of files changed by this merge request. You can - [configure the display](changes.md). +To view all merge requests for a project: -## View merge requests +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Merge requests**. + +Or, to use a [keyboard shortcut](../../shortcuts.md), press <kbd>g</kbd> + <kbd>m</kbd>. + +### View merge requests for all projects in a group -To view a list of merge requests: +To view merge requests for all projects in a group: -- **Merge requests for a project**: Go to your project and select **Merge requests**, or use - the <kbd>g</kbd> + <kbd>m</kbd> [keyboard shortcut](../../shortcuts.md) from a page in your project. -- **All projects in a group**: Go to your group and select **Merge requests**. - If your group contains subgroups, this view also displays merge requests from the subgroup projects. - GitLab displays a count of open merge requests in the left sidebar, but - [caches the value](reviews/index.md#cached-merge-request-count) for groups with a large number of - open merge requests. -- **Merge requests assigned to you**: On any GitLab page, select **Merge requests** - in the top bar, or use the <kbd>Shift</kbd> + <kbd>m</kbd> - [global keyboard shortcut](../../shortcuts.md). +1. On the top bar, select **Menu > Groups** and find your group. +1. On the left sidebar, select **Merge requests**. -GitLab displays open merge requests, with tabs to filter the list by open and closed status: +If your group contains subgroups, this view also displays merge requests from the subgroup projects. - +## View all merge requests assigned to you + +To view all merge requests assigned to you: + +1. On the top bar, put your cursor in the **Search** box. +1. From the dropdown list, select **Merge requests assigned to me**. + +Or, to use a [keyboard shortcut](../../shortcuts.md), press <kbd>Shift</kbd> + <kbd>m</kbd>. You can [search and filter](../../search/index.md#filter-issue-and-merge-request-lists), the results, or select a merge request to begin a review. -## Merge request sidebar - -The **Overview** tab of a merge request displays a sidebar. In this sidebar, you -can assign, categorize, and track progress on a merge request: - -- [**Assignee**](getting_started.md#assignee): Designate the directly responsible - individual (DRI) for a merge request. With - [multiple assignees](getting_started.md#multiple-assignees), you can assign a - merge request to more than one person at a time. -- [**Reviewer**](reviews/index.md): Designate a team member to review a merge request. - Higher tiers can assign multiple reviewers, and [require approvals](approvals/index.md) - from these reviewers. -- [**Milestone**](../milestones/index.md): Track time-sensitive changes. -- [**Time tracking**](../time_tracking.md): Time spent on a merge request. -- [**Labels**](../labels.md): Categorize a merge request and display it on - appropriate [issue boards](../issue_board.md). -- **Participants**: A list of users participating or watching a merge request. -- [**Notifications**](../../profile/notifications.md): A toggle to select whether - or not to receive notifications for updates to a merge request. - ## Add changes to a merge request If you have permission to add changes to a merge request, you can add your changes -to an existing merge request in several ways, depending on the complexity of your change and whether you need access to a development environment: +to an existing merge request in several ways, depending on the complexity of your +change and whether you need access to a development environment: -- [Edit changes in the Web IDE](../web_ide/index.md) in your browser. Use this +- [Edit changes in the Web IDE](../web_ide/index.md) in your browser with the + <kbd>.</kbd> [keyboard shortcut](../../shortcuts.md). Use this browser-based method to edit multiple files, or if you are not comfortable with Git commands. You cannot run tests from the Web IDE. - [Edit changes in Gitpod](../../../integration/gitpod.md#launch-gitpod-in-gitlab), if you @@ -158,3 +133,7 @@ For a web developer writing a webpage for your company's website: - [Authorization for merge requests](authorization_for_merge_requests.md) - [Testing and reports](testing_and_reports_in_merge_requests.md) - [GitLab keyboard shortcuts](../../shortcuts.md) +- [Comments and threads](../../discussions/index.md) +- [Suggest code changes](reviews/suggestions.md) +- [Commits](commits.md) +- [CI/CD pipelines](../../../ci/index.md) diff --git a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md index d7c9c0f73ee..256dde4fa17 100644 --- a/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md +++ b/doc/user/project/merge_requests/merge_when_pipeline_succeeds.md @@ -101,7 +101,7 @@ for details on avoiding two pipelines for a single merge request. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/211482) in GitLab 13.1. -When the **Pipelines must succeed** checkbox is checked, [skipped pipelines](../../../ci/yaml/index.md#skip-pipeline) prevent +When the **Pipelines must succeed** checkbox is checked, [skipped pipelines](../../../ci/pipelines/index.md#skip-a-pipeline) prevent merge requests from being merged. To change this behavior: 1. Navigate to your project's **Settings > General** page. diff --git a/doc/user/project/merge_requests/resolve_conflicts.md b/doc/user/project/merge_requests/resolve_conflicts.md index 4681ef09388..611f37a949b 100644 --- a/doc/user/project/merge_requests/resolve_conflicts.md +++ b/doc/user/project/merge_requests/resolve_conflicts.md @@ -1,85 +1,9 @@ --- -stage: Create -group: Code Review -info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments -type: reference, concepts +redirect_to: 'conflicts.md' +remove_date: '2022-01-26' --- -# Merge request conflict resolution **(FREE)** +This document was moved to [another location](conflicts.md). -Merge conflicts occur when two branches have different changes that cannot be -merged automatically. - -Git can merge changes between branches in most cases, but -occasionally Git requires your assistance to resolve the -conflicts manually. Typically, this is necessary when people change the same -parts of the same files. - -GitLab prevents merge requests from being merged until all conflicts are -resolved. Conflicts can be resolved locally, or in many cases in GitLab -(see [conflicts available for resolution](#conflicts-available-for-resolution) -for information on when this is available). - - - -NOTE: -GitLab resolves conflicts by creating a merge commit in the source branch that -is not automatically merged into the target branch. The merge -commit can be reviewed and tested before the changes are merged. This prevents -unintended changes entering the target branch without review or breaking the -build. - -## Resolve conflicts: interactive mode - -Clicking **Resolve Conflicts** displays a list of files with conflicts, with conflict sections -highlighted: - - - -After all conflicts have been marked as using 'ours' or 'theirs', the conflict -can be resolved. Resolving conflicts merges the target branch of the merge -request into the source branch, using the options -chosen. If the source branch is `feature` and the target branch is `main`, -this is similar to performing `git checkout feature; git merge main` locally. - -## Resolve conflicts: inline editor - -Some merge conflicts are more complex, requiring you to manually modify a file to -resolve them. Use the merge conflict resolution editor to resolve complex -conflicts in the GitLab interface. Click **Edit inline** to open the editor. -After you're sure about your changes, click **Commit to source branch**. - - - -## Conflicts available for resolution - -GitLab allows resolving conflicts in a file where all of the below are true: - -- The file is text, not binary -- The file is in a UTF-8 compatible encoding -- The file does not already contain conflict markers -- The file, with conflict markers added, is not over 200 KB in size -- The file exists under the same path in both branches - -If any file in your merge request containing conflicts can't meet all of these -criteria, you can't resolve the merge conflict in the UI. - -Additionally, GitLab does not detect conflicts in renames away from a path. For -example, this does not create a conflict: - -1. On branch `a`, doing `git mv file1 file2` -1. On branch `b`, doing `git mv file1 file3`. - -Instead, both files are present in the branch after the merge request is merged. - -<!-- ## Troubleshooting - -Include any troubleshooting steps that you can foresee. If you know beforehand what issues -one might have when setting this up, or when something is changed, or on upgrading, it's -important to describe those, too. Think of things that may go wrong and include them here. -This is important to minimize requests for support, and to avoid doc comments with -questions that you know someone might ask. - -Each scenario can be a third-level heading, e.g. `### Getting error message X`. -If you have none to add when creating a doc, leave this section in place -but commented out to help encourage others to add to it in the future. --> +<!-- This redirect file can be deleted after <2022-01-26>. --> +<!-- Before deletion, see: https://docs.gitlab.com/ee/development/documentation/#move-or-rename-a-page --> diff --git a/doc/user/project/merge_requests/reviews/index.md b/doc/user/project/merge_requests/reviews/index.md index e6f84f1c357..597dcb3dfb9 100644 --- a/doc/user/project/merge_requests/reviews/index.md +++ b/doc/user/project/merge_requests/reviews/index.md @@ -17,6 +17,10 @@ your merge request, and makes [code suggestions](suggestions.md) you can accept from the user interface. When your work is reviewed, your team members can choose to accept or reject it. +You can review merge requests from the GitLab interface. If you install the +[GitLab Workflow VS Code extension](../../repository/vscode.md), you can also +review merge requests in Visual Studio Code. + ## Review a merge request > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/4213) in GitLab Premium 11.4. @@ -207,7 +211,7 @@ These features are associated with merge requests: When viewing the commit details page, GitLab links to the merge request(s) containing that commit. - [Merge requests versions](../versions.md): Select and compare the different versions of merge request diffs -- [Resolve conflicts](../resolve_conflicts.md): +- [Resolve conflicts](../conflicts.md): GitLab can provide the option to resolve certain merge request conflicts in the GitLab UI. - [Revert changes](../revert_changes.md): Revert changes from any commit from a merge request. diff --git a/doc/user/project/merge_requests/reviews/suggestions.md b/doc/user/project/merge_requests/reviews/suggestions.md index ac1cd57fb99..7bfb8e52279 100644 --- a/doc/user/project/merge_requests/reviews/suggestions.md +++ b/doc/user/project/merge_requests/reviews/suggestions.md @@ -7,14 +7,13 @@ type: index, reference # Suggest changes **(FREE)** -> - [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/18008) in GitLab 11.6. -> - Custom commit messages for suggestions was [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25381) in GitLab 13.9 behind a [feature flag](../../../feature_flags.md), disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25381) custom commit messages for suggestions in GitLab 13.9 [with a flag](../../../../administration/feature_flags.md) named `suggestions_custom_commit`. Disabled by default. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/297404) in GitLab 13.10. As a reviewer, you're able to suggest code changes with a Markdown syntax in merge request diff threads. Then, the merge request author (or other users with appropriate -[permission](../../../permissions.md)) is able to apply these suggestions with a click, -which generates a commit in the merge request authored by the user that suggested the changes. +[permission](../../../permissions.md)) can apply these suggestions with a click. +This action generates a commit in the merge request, authored by the user that suggested the changes. 1. Choose a line of code to be changed, add a new comment, then select the **Insert suggestion** icon in the toolbar: @@ -38,14 +37,15 @@ which generates a commit in the merge request authored by the user that suggeste  -After the author applies a suggestion, it's marked with the **Applied** label, -the thread is automatically resolved, and GitLab creates a new commit -and pushes the suggested change directly into the codebase in the merge request's -branch. The [Developer role](../../../permissions.md) is required to do so. +After the author applies a suggestion: -## Multi-line suggestions +1. The suggestion is marked as **Applied**. +1. The thread is resolved. +1. GitLab creates a new commit with the changes. +1. If the user has the [Developer role](../../../permissions.md), GitLab pushes + the suggested change directly into the codebase in the merge request's branch. -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/53310) in GitLab 11.10. +## Multi-line suggestions Reviewers can also suggest changes to multiple lines with a single suggestion within merge request diff threads by adjusting the range offsets. The @@ -67,9 +67,9 @@ suggestion. ## Code block nested in suggestions -If you need to make a suggestion that involves a -[fenced code block](../../../markdown.md#code-spans-and-blocks), wrap your suggestion in four backticks -instead of the usual three. +To add a suggestion that includes a +[fenced code block](../../../markdown.md#code-spans-and-blocks), wrap your suggestion +in four backticks instead of three:  @@ -95,14 +95,14 @@ You can also use following variables besides static text: | Variable | Description | Output example | |------------------------|-------------|----------------| -| `%{branch_name}` | The name of the branch the suggestion(s) was(were) applied to. | `my-feature-branch` | -| `%{files_count}` | The number of file(s) to which suggestion(s) was(were) applied.| **2** | -| `%{file_paths}` | The path(s) of the file(s) suggestion(s) was(were) applied to. Paths are separated by commas.| `docs/index.md, docs/about.md` | +| `%{branch_name}` | The name of the branch to which suggestions were applied. | `my-feature-branch` | +| `%{files_count}` | The number of files to which suggestions were applied.| **2** | +| `%{file_paths}` | The paths of the file to which suggestions were applied. Paths are separated by commas.| `docs/index.md, docs/about.md` | | `%{project_path}` | The project path. | `my-group/my-project` | | `%{project_name}` | The human-readable name of the project. | **My Project** | | `%{suggestions_count}` | The number of suggestions applied.| **3** | -| `%{username}` | The username of the user applying suggestion(s). | `user_1` | -| `%{user_full_name}` | The full name of the user applying suggestion(s). | **User 1** | +| `%{username}` | The username of the user applying suggestions. | `user_1` | +| `%{user_full_name}` | The full name of the user applying suggestions. | **User 1** | For example, to customize the commit message to output **Addresses user_1's review**, set the custom text to @@ -114,32 +114,32 @@ introduced by [#25381](https://gitlab.com/gitlab-org/gitlab/-/issues/25381). ## Batch suggestions -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25486) in GitLab 13.1 as an [alpha feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha) behind a feature flag, disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/25486) in GitLab 13.1 as an [alpha feature](https://about.gitlab.com/handbook/product/gitlab-the-product/#alpha) with a flag named `batch_suggestions`, disabled by default. > - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/227799) in GitLab 13.2. -> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/320755) in GitLab 13.11. -> - Custom commit messages for batch suggestions [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326168) in GitLab 14.4. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/320755) in GitLab 13.11. [Feature flag `batch_suggestions`](https://gitlab.com/gitlab-org/gitlab/-/issues/320755) removed. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/326168) custom commit messages for batch suggestions in GitLab 14.4. You can apply multiple suggestions at once to reduce the number of commits added to your branch to address your reviewers' requests. 1. To start a batch of suggestions to apply with a single commit, select **Add suggestion to batch**: -  +  1. Add as many additional suggestions to the batch as you wish: -  +  1. To remove suggestions, select **Remove from batch**: -  +  1. Having added all the suggestions to your liking, when ready, select **Apply suggestions**. You can optionally specify a custom commit message for [batch suggestions](#batch-suggestions) (GitLab 14.4 and later) to describe your change. If you don't specify it, the default commit message is used. -  +  WARNING: Suggestions applied from multiple authors creates a commit authored by the user applying the suggestions. diff --git a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md index 27487003697..ee4320d5ea1 100644 --- a/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md +++ b/doc/user/project/pages/custom_domains_ssl_tls_certification/index.md @@ -27,6 +27,17 @@ and steps below. - Access to your domain's server control panel to set up DNS records: - A DNS A or CNAME record pointing your domain to GitLab Pages server. - A DNS `TXT` record to verify your domain's ownership. +- Set either `external_http` or `external_https` in `/etc/gitlab/gitlab.rb` to the IP and port of + your [Pages Daemon](../../../../administration/pages/index.md#overview). + If you don't have IPv6, you can omit the IPv6 address. + + Example: + + ```ruby + # Redirect pages from HTTP to HTTPS + gitlab_pages['external_http'] = ['192.0.2.2:80', '[2001:db8::2]:80'] # The secondary IPs for the GitLab Pages daemon + gitlab_pages['external_https'] = ['192.0.2.2:443', '[2001:db8::2]:443'] # The secondary IPs for the GitLab Pages daemon + ``` ### Steps diff --git a/doc/user/project/pages/getting_started/pages_ci_cd_template.md b/doc/user/project/pages/getting_started/pages_ci_cd_template.md index 4f2b62beab1..25382778b1d 100644 --- a/doc/user/project/pages/getting_started/pages_ci_cd_template.md +++ b/doc/user/project/pages/getting_started/pages_ci_cd_template.md @@ -13,31 +13,16 @@ the CI/CD pipeline to generate a Pages website. Use a `.gitlab-ci.yml` template when you have an existing project that you want to add a Pages site to. -Your GitLab repository should contain files specific to an SSG, or plain HTML. -After you complete these steps, you may need to do additional -configuration for the Pages site to generate properly. - -1. On the left sidebar, select **Project information**. -1. Click **Set up CI/CD**. - -  - - If this button is not available, CI/CD is already configured for - your project. You may want to browse the `.gitlab-ci.yml` files - [in these projects instead](https://gitlab.com/pages). - -1. From the **Apply a template** list, choose a template for the SSG you're using. - You can also choose plain HTML. - -  - - If you don't find a corresponding template, you can view the - [GitLab Pages group of sample projects](https://gitlab.com/pages). - These projects contain `.gitlab-ci.yml` files that you can modify for your needs. - You can also [learn how to write your own `.gitlab-ci.yml` - file for GitLab Pages](pages_from_scratch.md). - -1. Save and commit the `.gitlab-ci.yml` file. +Your GitLab repository should contain files specific to an SSG, or plain HTML. After you complete +these steps, you may have to do additional configuration for the Pages site to generate properly. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select the project's name. +1. From the **Add** (**{plus}**) dropdown, select **New file**. +1. From the **Select a template type** dropdown, select `.gitlab-ci.yml`. +1. From the **Apply a template** dropdown, in the **Pages** section, select the name of your SSG. +1. In the **Commit message** box, type the commit message. +1. Select **Commit changes**. If everything is configured correctly, the site can take approximately 30 minutes to deploy. diff --git a/doc/user/project/pages/img/choose_ci_template_v13_1.png b/doc/user/project/pages/img/choose_ci_template_v13_1.png Binary files differdeleted file mode 100644 index a0c25ba1642..00000000000 --- a/doc/user/project/pages/img/choose_ci_template_v13_1.png +++ /dev/null diff --git a/doc/user/project/pages/img/setup_ci_v13_1.png b/doc/user/project/pages/img/setup_ci_v13_1.png Binary files differdeleted file mode 100644 index 2cf1c05d6d8..00000000000 --- a/doc/user/project/pages/img/setup_ci_v13_1.png +++ /dev/null diff --git a/doc/user/project/pages/introduction.md b/doc/user/project/pages/introduction.md index 45c2f1aaf04..59a2f0c2eba 100644 --- a/doc/user/project/pages/introduction.md +++ b/doc/user/project/pages/introduction.md @@ -223,16 +223,13 @@ Consider a Pages site deployed with the following files: ```plaintext public/ -├─┬ index.html -│ ├ data.html -│ └ info.html -│ +├── index.html +├── data.html +├── info.html ├── data/ │ └── index.html -├── info/ -│ └── details.html -└── other/ - └── index.html +└── info/ + └── details.html ``` Pages supports reaching each of these files through several different URLs. In @@ -241,23 +238,19 @@ specifies the directory. If the URL references a file that doesn't exist, but adding `.html` to the URL leads to a file that *does* exist, it's served instead. Here are some examples of what happens given the above Pages site: -| URL path | HTTP response | File served | -| -------------------- | ------------- | ----------- | -| `/` | `200 OK` | `public/index.html` | -| `/index.html` | `200 OK` | `public/index.html` | -| `/index` | `200 OK` | `public/index.html` | -| `/data` | `200 OK` | `public/data/index.html` | -| `/data/` | `200 OK` | `public/data/index.html` | -| `/data.html` | `200 OK` | `public/data.html` | -| `/info` | `200 OK` | `public/info.html` | -| `/info/` | `200 OK` | `public/info.html` | -| `/info.html` | `200 OK` | `public/info.html` | -| `/info/details` | `200 OK` | `public/info/details.html` | -| `/info/details.html` | `200 OK` | `public/info/details.html` | -| `/other` | `302 Found` | `public/other/index.html` | -| `/other/` | `200 OK` | `public/other/index.html` | -| `/other/index` | `200 OK` | `public/other/index.html` | -| `/other/index.html` | `200 OK` | `public/other/index.html` | +| URL path | HTTP response | +| -------------------- | ------------- | +| `/` | `200 OK`: `public/index.html` | +| `/index.html` | `200 OK`: `public/index.html` | +| `/index` | `200 OK`: `public/index.html` | +| `/data` | `302 Found`: redirecting to `/data/` | +| `/data/` | `200 OK`: `public/data/index.html` | +| `/data.html` | `200 OK`: `public/data.html` | +| `/info` | `302 Found`: redirecting to `/info/` | +| `/info/` | `404 Not Found` Error Page | +| `/info.html` | `200 OK`: `public/info.html` | +| `/info/details` | `200 OK`: `public/info/details.html` | +| `/info/details.html` | `200 OK`: `public/info/details.html` | Note that when `public/data/index.html` exists, it takes priority over the `public/data.html` file for both the `/data` and `/data/` URL paths. diff --git a/doc/user/project/push_options.md b/doc/user/project/push_options.md index 305bbb2ded0..15df69ee68c 100644 --- a/doc/user/project/push_options.md +++ b/doc/user/project/push_options.md @@ -69,8 +69,8 @@ time as pushing changes: | `merge_request.milestone="<milestone>"` | Set the milestone of the merge request. Ex: `git push -o merge_request.milestone="3.0"`. | [14.1](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/63960) | | `merge_request.label="<label>"` | Add labels to the merge request. If the label does not exist, it is created. For example, for two labels: `git push -o merge_request.label="label1" -o merge_request.label="label2"`. | [12.3](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31831) | | `merge_request.unlabel="<label>"` | Remove labels from the merge request. For example, for two labels: `git push -o merge_request.unlabel="label1" -o merge_request.unlabel="label2"`. | [12.3](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/31831) | -| `merge_request.assign="<user>"` | Assign users to the merge request. For example, for two users: `git push -o merge_request.assign="user1" -o merge_request.assign="user2"`. | [13.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25904) | -| `merge_request.unassign="<user>"` | Remove assigned users from the merge request. For example, for two users: `git push -o merge_request.unassign="user1" -o merge_request.unassign="user2"`. | [13.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25904) | +| `merge_request.assign="<user>"` | Assign users to the merge request. Accepts username or user ID. For example, for two users: `git push -o merge_request.assign="user1" -o merge_request.assign="user2"`. | [13.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25904) | +| `merge_request.unassign="<user>"` | Remove assigned users from the merge request. Accepts username or user ID.For example, for two users: `git push -o merge_request.unassign="user1" -o merge_request.unassign="user2"`. | [13.10](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/25904) | If you use a push option that requires text with spaces in it, you need to enclose it in quotes (`"`). You can omit the quotes if there are no spaces. Some examples: @@ -117,3 +117,13 @@ pipeline succeeds: ```shell git mwps origin <local-branch-name> ``` + +## Troubleshooting + +## Push options for merge request assignment ignored + +When you push a branch to GitLab, you can use push options to assign to (`merge_request.assign="<USERNAME>"`) +or unassign from (`merge_request.unassign="<USERNAME>"`) a user. If GitLab creates +the merge request successfully, but fails to assign or unassign the merge request +correctly, you can use the user ID instead. For more information, read the issue +[Push option `merge_request.(un)assign` seems to be ignored](https://gitlab.com/gitlab-org/gitlab/-/issues/325169). diff --git a/doc/user/project/quick_actions.md b/doc/user/project/quick_actions.md index 45a83394c41..a4d7b94506b 100644 --- a/doc/user/project/quick_actions.md +++ b/doc/user/project/quick_actions.md @@ -78,6 +78,7 @@ threads. Some quick actions might not be available to all subscription tiers. | `/move <path/to/project>` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Move this issue to another project. | | `/parent_epic <epic>` | **{dotted-circle}** No | **{dotted-circle}** No | **{check-circle}** Yes | Set parent epic to `<epic>`. The `<epic>` value should be in the format of `&epic`, `group&epic`, or a URL to an epic ([introduced in GitLab 12.1](https://gitlab.com/gitlab-org/gitlab/-/issues/10556)). | | `/promote` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Promote issue to epic. | +| `/promote_to_incident` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Promote issue to incident ([introduced in GitLab 14.5](https://gitlab.com/gitlab-org/gitlab/-/issues/296787)). | | `/publish` | **{check-circle}** Yes | **{dotted-circle}** No | **{dotted-circle}** No | Publish issue to an associated [Status Page](../../operations/incident_management/status_page.md) ([Introduced in GitLab 13.0](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/30906)) | | `/reassign @user1 @user2` | **{check-circle}** Yes | **{check-circle}** Yes | **{dotted-circle}** No | Replace current assignees with those specified. | | `/rebase` | **{dotted-circle}** No | **{check-circle}** Yes | **{dotted-circle}** No | Rebase source branch. This schedules a background task that attempts to rebase the changes in the source branch on the latest commit of the target branch. If `/rebase` is used, `/merge` is ignored to avoid a race condition where the source branch is merged or deleted before it is rebased. If there are merge conflicts, GitLab displays a message that a rebase cannot be scheduled. Rebase failures are displayed with the merge request status. | diff --git a/doc/user/project/releases/index.md b/doc/user/project/releases/index.md index 9e4d621dbc6..abedae5d10b 100644 --- a/doc/user/project/releases/index.md +++ b/doc/user/project/releases/index.md @@ -78,17 +78,172 @@ To create a new release through the GitLab UI: [release notes](#release-notes-description), or [assets links](#links). 1. Click **Create release**. -### Create release from GitLab CI +## Create a release by using a CI/CD job > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/19298) in GitLab 12.7. -You can [create a release directly from the GitLab CI pipeline](../../../ci/yaml/index.md#release) -by using a `release` node in the job definition. +You can create a release directly as part of the GitLab CI/CD pipeline +by using [the `release` keyword](../../../ci/yaml/index.md#release) in the job definition. The release is created only if the job processes without error. If the Rails API returns an error during release creation, the release job fails. -### Upcoming releases +### CI/CD example of the `release` keyword + +To create a release when you push a Git tag, or when you add a Git tag +in the UI by going to **Repository > Tags**: + +```yaml +release_job: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + rules: + - if: $CI_COMMIT_TAG # Run this job when a tag is created manually + script: + - echo 'running release_job' + release: + name: 'Release $CI_COMMIT_TAG' + description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION must be defined + tag_name: '$CI_COMMIT_TAG' # elsewhere in the pipeline. + ref: '$CI_COMMIT_TAG' + milestones: + - 'm1' + - 'm2' + - 'm3' + released_at: '2020-07-15T08:00:00Z' # Optional, is auto generated if not defined, or can use a variable. + assets: # Optional, multiple asset links + links: + - name: 'asset1' + url: 'https://example.com/assets/1' + - name: 'asset2' + url: 'https://example.com/assets/2' + filepath: '/pretty/url/1' # optional + link_type: 'other' # optional +``` + +To create a release automatically when commits are pushed or merged to the default branch, +using a new Git tag that is defined with variables: + +```yaml +prepare_job: + stage: prepare # This stage must run before the release stage + rules: + - if: $CI_COMMIT_TAG + when: never # Do not run this job when a tag is created manually + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch + script: + - echo "EXTRA_DESCRIPTION=some message" >> variables.env # Generate the EXTRA_DESCRIPTION and TAG environment variables + - echo "TAG=v$(cat VERSION)" >> variables.env # and append to the variables.env file + artifacts: + reports: + dotenv: variables.env # Use artifacts:reports:dotenv to expose the variables to other jobs + +release_job: + stage: release + image: registry.gitlab.com/gitlab-org/release-cli:latest + needs: + - job: prepare_job + artifacts: true + rules: + - if: $CI_COMMIT_TAG + when: never # Do not run this job when a tag is created manually + - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH # Run this job when commits are pushed or merged to the default branch + script: + - echo 'running release_job for $TAG' + release: + name: 'Release $TAG' + description: 'Created using the release-cli $EXTRA_DESCRIPTION' # $EXTRA_DESCRIPTION and the $TAG + tag_name: '$TAG' # variables must be defined elsewhere + ref: '$CI_COMMIT_SHA' # in the pipeline. For example, in the + milestones: # prepare_job + - 'm1' + - 'm2' + - 'm3' + released_at: '2020-07-15T08:00:00Z' # Optional, is auto generated if not defined, or can use a variable. + assets: + links: + - name: 'asset1' + url: 'https://example.com/assets/1' + - name: 'asset2' + url: 'https://example.com/assets/2' + filepath: '/pretty/url/1' # optional + link_type: 'other' # optional +``` + +NOTE: +Environment variables set in `before_script` or `script` are not available for expanding +in the same job. Read more about +[potentially making variables available for expanding](https://gitlab.com/gitlab-org/gitlab-runner/-/issues/6400). + +### Use a custom SSL CA certificate authority + +You can use the `ADDITIONAL_CA_CERT_BUNDLE` CI/CD variable to configure a custom SSL CA certificate authority, +which is used to verify the peer when the `release-cli` creates a release through the API using HTTPS with custom certificates. +The `ADDITIONAL_CA_CERT_BUNDLE` value should contain the +[text representation of the X.509 PEM public-key certificate](https://tools.ietf.org/html/rfc7468#section-5.1) +or the `path/to/file` containing the certificate authority. +For example, to configure this value in the `.gitlab-ci.yml` file, use the following: + +```yaml +release: + variables: + ADDITIONAL_CA_CERT_BUNDLE: | + -----BEGIN CERTIFICATE----- + MIIGqTCCBJGgAwIBAgIQI7AVxxVwg2kch4d56XNdDjANBgkqhkiG9w0BAQsFADCB + ... + jWgmPqF3vUbZE0EyScetPJquRFRKIesyJuBFMAs= + -----END CERTIFICATE----- + script: + - echo "Create release" + release: + name: 'My awesome release' + tag_name: '$CI_COMMIT_TAG' +``` + +The `ADDITIONAL_CA_CERT_BUNDLE` value can also be configured as a +[custom variable in the UI](../../../ci/variables/index.md#custom-cicd-variables), +either as a `file`, which requires the path to the certificate, or as a variable, +which requires the text representation of the certificate. + +### `release-cli` command line + +The entries under the `release` node are transformed into a `bash` command line and sent +to the Docker container, which contains the [release-cli](https://gitlab.com/gitlab-org/release-cli). +You can also call the `release-cli` directly from a `script` entry. + +For example, if you use the YAML described previously: + +```shell +release-cli create --name "Release $CI_COMMIT_SHA" --description "Created using the release-cli $EXTRA_DESCRIPTION" --tag-name "v${MAJOR}.${MINOR}.${REVISION}" --ref "$CI_COMMIT_SHA" --released-at "2020-07-15T08:00:00Z" --milestone "m1" --milestone "m2" --milestone "m3" --assets-link "{\"name\":\"asset1\",\"url\":\"https://example.com/assets/1\",\"link_type\":\"other\"} +``` + +### Create multiple releases in a single pipeline + +A pipeline can have multiple `release` jobs, for example: + +```yaml +ios-release: + script: + - echo 'iOS release job' + release: + tag_name: v1.0.0-ios + description: 'iOS release v1.0.0' + +android-release: + script: + - echo 'Android release job' + release: + tag_name: v1.0.0-android + description: 'Android release v1.0.0' +``` + +### Release assets as Generic packages + +You can use [Generic packages](../../packages/generic_packages/index.md) to host your release assets. +For a complete example, see the [Release assets as Generic packages](https://gitlab.com/gitlab-org/release-cli/-/tree/master/docs/examples/release-assets-as-generic-package/) +project. + +## Upcoming releases > [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/38105) in GitLab 12.1. @@ -238,14 +393,6 @@ The release title can be customized using the **Release title** field when creating or editing a release. If no title is provided, the release's tag name is used instead. -Guest users of private projects are allowed to view the **Releases** page -but are _not_ allowed to view details about the Git repository (in particular, -tag names). Because of this, release titles are replaced with a generic -title like "Release-1234" for Guest users to avoid leaking tag name information. - -See the [Release permissions](#release-permissions) section for -more information about permissions. - ### Tag name The release tag name should include the release version. GitLab uses [Semantic Versioning](https://semver.org/) @@ -569,11 +716,14 @@ In the API: ### View a release and download assets -- Users with [Reporter role or above](../../../user/permissions.md#project-members-permissions) +> [Changes were made to the Guest role access](https://gitlab.com/gitlab-org/gitlab/-/issues/335209) in GitLab 14.5. + +- Users with the [Reporter role or above](../../../user/permissions.md#project-members-permissions) + have read and download access to the project releases. +- Users with the [Guest role](../../../user/permissions.md#project-members-permissions) have read and download access to the project releases. -- Users with [Guest role](../../../user/permissions.md#project-members-permissions) - have read and download access to the project releases, however, - repository-related information are redacted (for example the Git tag name). + This includes associated Git-tag-names, release description, author information of the releases. + However, other repository-related information, such as [source code](#source-code), [release evidence](#release-evidence) are redacted. ### Create, update, and delete a release and its assets diff --git a/doc/user/project/releases/release_cli.md b/doc/user/project/releases/release_cli.md new file mode 100644 index 00000000000..c23ceaa1aba --- /dev/null +++ b/doc/user/project/releases/release_cli.md @@ -0,0 +1,76 @@ +--- +type: reference, howto +stage: Release +group: Release +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Install the `release-cli` for the Shell executor + +> - [Introduced](https://gitlab.com/gitlab-org/release-cli/-/issues/21) in GitLab 13.8. +> - [Changed](https://gitlab.com/gitlab-org/release-cli/-/merge_requests/108) in GitLab 14.2, the `release-cli` binaries are also [available in the Package Registry](https://gitlab.com/jaime/release-cli/-/packages). + +When you use a runner with the Shell executor, you can download and install +the `release-cli` manually for your [supported OS and architecture](https://release-cli-downloads.s3.amazonaws.com/latest/index.html). +Once installed, [the `release` keyword](../../../ci/yaml/index.md#release) is available to use in your CI/CD jobs. + +## Install on Unix/Linux + +1. Download the binary for your system from S3, in the following example for amd64 systems: + + ```shell + curl --location --output /usr/local/bin/release-cli "https://release-cli-downloads.s3.amazonaws.com/latest/release-cli-linux-amd64" + ``` + + Or from the GitLab Package Registry: + + ```shell + curl --location --output /usr/local/bin/release-cli "https://gitlab.com/api/v4/projects/gitlab-org%2Frelease-cli/packages/generic/release-cli/latest/release-cli-darwin-amd64" + ``` + +1. Give it permissions to execute: + + ```shell + sudo chmod +x /usr/local/bin/release-cli + ``` + +1. Verify `release-cli` is available: + + ```shell + $ release-cli -v + + release-cli version 0.6.0 + ``` + +## Install on Windows PowerShell + +1. Create a folder somewhere in your system, for example `C:\GitLab\Release-CLI\bin` + + ```shell + New-Item -Path 'C:\GitLab\Release-CLI\bin' -ItemType Directory + ``` + +1. Download the executable file: + + ```shell + PS C:\> Invoke-WebRequest -Uri "https://release-cli-downloads.s3.amazonaws.com/latest/release-cli-windows-amd64.exe" -OutFile "C:\GitLab\Release-CLI\bin\release-cli.exe" + + Directory: C:\GitLab\Release-CLI + Mode LastWriteTime Length Name + ---- ------------- ------ ---- + d----- 3/16/2021 4:17 AM bin + ``` + +1. Add the directory to your `$env:PATH`: + + ```shell + $env:PATH += ";C:\GitLab\Release-CLI\bin" + ``` + +1. Verify `release-cli` is available: + + ```shell + PS C:\> release-cli -v + + release-cli version 0.6.0 + ``` diff --git a/doc/user/project/repository/branches/default.md b/doc/user/project/repository/branches/default.md index 5cd025f017d..bc6bc799670 100644 --- a/doc/user/project/repository/branches/default.md +++ b/doc/user/project/repository/branches/default.md @@ -167,7 +167,7 @@ changed, GitLab records the name of the old default branch. If that branch is deleted, attempts to view a file or directory on it are redirected to the current default branch, instead of displaying the "not found" page. -## Resources +## Related topics - [Configure a default branch for your wiki](../../wiki/index.md) - [Discussion of default branch renaming](https://lore.kernel.org/git/pull.656.v4.git.1593009996.gitgitgadget@gmail.com/) diff --git a/doc/user/project/repository/gpg_signed_commits/index.md b/doc/user/project/repository/gpg_signed_commits/index.md index 23d7aecc960..6aa32b1b816 100644 --- a/doc/user/project/repository/gpg_signed_commits/index.md +++ b/doc/user/project/repository/gpg_signed_commits/index.md @@ -39,7 +39,10 @@ For a commit to be verified by GitLab: - The committer's public key must have been uploaded to their GitLab account. - One of the emails in the GPG key must match a **verified** email address - used by the committer in GitLab. + used by the committer in GitLab. This address will be part of the public key. + If you want to keep this address private, use the automatically generated + [private commit email address](../../../profile/index.md#use-an-automatically-generated-private-commit-email) + GitLab provides in your profile. - The committer's email address must match the verified email address from the GPG key. diff --git a/doc/user/project/repository/index.md b/doc/user/project/repository/index.md index de7459e6278..bb1a55f6b2b 100644 --- a/doc/user/project/repository/index.md +++ b/doc/user/project/repository/index.md @@ -41,7 +41,7 @@ to a branch in the repository. When you use the command line, you can commit mul If the project is configured with [GitLab CI/CD](../../../ci/index.md), you trigger a pipeline per push, not per commit. - **Skip pipelines:** - Add the [`ci skip`](../../../ci/yaml/index.md#skip-pipeline) keyword to + Add the [`ci skip`](../../../ci/pipelines/index.md#skip-a-pipeline) keyword to your commit message to make GitLab CI/CD skip the pipeline. - **Cross-link issues and merge requests:** Use [cross-linking](../issues/crosslinking_issues.md#from-commit-messages) @@ -82,14 +82,19 @@ prompted to open XCode. > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/220957) in GitLab 13.10. -All projects can be cloned into Visual Studio Code. To do that: +All projects can be cloned into Visual Studio Code from the GitLab user interface, but you +can also install the [GitLab Workflow VS Code extension](vscode.md) to clone from +Visual Studio Code: -1. From the GitLab UI, go to the project's overview page. -1. Click **Clone**. -1. Select **Clone with Visual Studio Code** under either HTTPS or SSH method. -1. Select a folder to clone the project into. +- From the GitLab interface: + 1. Go to the project's overview page. + 1. Select **Clone**. + 1. Under either the **HTTPS** or **SSH** method, select **Clone with Visual Studio Code**. + 1. Select a folder to clone the project into. -When VS Code has successfully cloned your project, it opens the folder. + After Visual Studio Code clones your project, it opens the folder. +- From Visual Studio Code, with the [extension](vscode.md) installed, use the + extension's [`Git: Clone` command](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#clone-gitlab-projects). ## Download the code in a repository @@ -243,6 +248,10 @@ When you [rename a user](../../profile/index.md#change-your-username), - The redirects are available as long as the original path is not claimed by another group, user, or project. +## Related links + +- [GitLab Workflow VS Code extension](vscode.md) + ## Troubleshooting ### Repository Languages: excessive CPU use diff --git a/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook_diff_v14_5.png b/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook_diff_v14_5.png Binary files differnew file mode 100644 index 00000000000..bc63322cd65 --- /dev/null +++ b/doc/user/project/repository/jupyter_notebooks/img/jupyter_notebook_diff_v14_5.png diff --git a/doc/user/project/repository/jupyter_notebooks/index.md b/doc/user/project/repository/jupyter_notebooks/index.md index 2ad1504aac3..d040cc93876 100644 --- a/doc/user/project/repository/jupyter_notebooks/index.md +++ b/doc/user/project/repository/jupyter_notebooks/index.md @@ -4,22 +4,42 @@ group: Source Code info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" type: reference --- -# Jupyter Notebook Files **(FREE)** +# Jupyter Notebook files **(FREE)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/2508/) in GitLab 9.1. +[Jupyter Notebook](https://jupyter.org/) (previously, IPython Notebook) files are used for +interactive computing in many fields. They contain a complete record of the +user's sessions and include: -[Jupyter](https://jupyter.org/) Notebook (previously IPython Notebook) files are used for -interactive computing in many fields and contain a complete record of the -user's sessions and include code, narrative text, equations, and rich output. +- Code. +- Narrative text. +- Equations. +- Rich output. -When added to a repository, Jupyter Notebooks with a `.ipynb` extension are -rendered to HTML when viewed: +When you add a Jupyter Notebook (with `.ipynb` extension) to your repository, +it's rendered into HTML when you view it:  Interactive features, including JavaScript plots, don't work when viewed in GitLab. +## Cleaner diffs + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/epics/6589) in GitLab 14.5 [with a flag](../../../../administration/feature_flags.md) named `jupyter_clean_diffs`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../../../../administration/feature_flags.md) named `jupyter_clean_diffs`. +On GitLab.com, this feature is available. + +When commits include changes to Jupyter Notebook files, GitLab: + +- Transforms the machine-readable `.ipynb` file into a human-readable Markdown file. +- Displays a cleaner version of the diff that includes syntax highlighting. + +Code suggestions are not available on diffs and merge requests for `.ipynb` files. + + + ## Jupyter Git integration Jupyter can be configured as an OAuth application with repository access, acting diff --git a/doc/user/project/repository/mirror/bidirectional.md b/doc/user/project/repository/mirror/bidirectional.md index c4254655fcc..340d7b48a47 100644 --- a/doc/user/project/repository/mirror/bidirectional.md +++ b/doc/user/project/repository/mirror/bidirectional.md @@ -139,7 +139,7 @@ This sample has a few limitations: - The script circumvents the Git hook quarantine environment because the update of `$TARGET_REPO` is seen as a ref update, and Git displays warnings about it. -## Mirror with Perforce Helix via Git Fusion **(PREMIUM)** +## Mirror with Perforce Helix with Git Fusion **(PREMIUM)** > Moved to GitLab Premium in 13.9. diff --git a/doc/user/project/repository/mirror/img/repository_mirroring_copy_ssh_public_key_button.png b/doc/user/project/repository/mirror/img/repository_mirroring_copy_ssh_public_key_button.png Binary files differdeleted file mode 100644 index e20dae09a4d..00000000000 --- a/doc/user/project/repository/mirror/img/repository_mirroring_copy_ssh_public_key_button.png +++ /dev/null diff --git a/doc/user/project/repository/mirror/index.md b/doc/user/project/repository/mirror/index.md index 4532a80c2f5..361c0902ebf 100644 --- a/doc/user/project/repository/mirror/index.md +++ b/doc/user/project/repository/mirror/index.md @@ -7,102 +7,162 @@ disqus_identifier: 'https://docs.gitlab.com/ee/workflow/repository_mirroring.htm # Repository mirroring **(FREE)** -Repository mirroring allows for the mirroring of repositories to and from external sources. You -can use it to mirror branches, tags, and commits between repositories. It helps you use -a repository outside of GitLab. +You can _mirror_ a repository to and from external sources. You can select which +repository serves as the source, and modify which parts of the repository are copied. +Branches, tags, and commits can be mirrored. -A repository mirror at GitLab updates automatically. You can also manually trigger an update: - -- At most once every five minutes on GitLab.com. -- According to a [limit set by the administrator](../../../../administration/instance_limits.md#pull-mirroring-interval) - on self-managed instances. - -There are two kinds of repository mirroring supported by GitLab: +Several mirroring methods exist: - [Push](push.md): for mirroring a GitLab repository to another location. - [Pull](pull.md): for mirroring a repository from another location to GitLab. +- [Bidirectional](bidirectional.md) mirroring is also available, but can cause conflicts. + +Mirror a repository when: + +- The canonical version of your project has migrated to GitLab. To keep providing a + copy of your project at its previous home, configure your GitLab repository as a + [push mirror](push.md). Changes you make to your GitLab repository are copied to + the old location. +- Your GitLab project is private, but some components can be shared publicly. + Configure your primary repository as a [push mirror](push.md) and push the portions + you want to make public. With this configuration, you can open-source specific + projects, contribute back to the open-source community, and protect the sensitive + parts of your project. +- You migrated to GitLab, but the canonical version of your project is somewhere else. + Configure your GitLab repository as a [pull mirror](pull.md) of the other project. + Your GitLab repository pulls copies of the commits, tags, and branches of project. + They become available to use on GitLab. + +## Create a repository mirror + +Prerequisite: + +- You must have at least the [Maintainer role](../../../permissions.md) for the project. +- If your mirror connects with `ssh://`, the host key must be detectable on the server, + or you must have a local copy of the key. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Mirroring repositories**. +1. Enter a **Git repository URL**. For security reasons, the URL to the original + repository is only displayed to users with the [Maintainer role](../../../permissions.md) + or the [Owner role](../../../permissions.md) for the mirrored project. +1. Select a **Mirror direction**. +1. If you entered a `ssh://` URL, select either: + - **Detect host keys**: GitLab fetches the host keys from the server and displays the fingerprints. + - **Input host keys manually**, and enter the host key into **SSH host key**. + + When mirroring the repository, GitLab confirms at least one of the stored host keys + matches before connecting. This check can protect your mirror from malicious code injections, + or your password from being stolen. +1. Select an **Authentication method**. To learn more, read + [Authentication methods for mirrors](#authentication-methods-for-mirrors). +1. If you authenticate with SSH host keys, [verify the host key](#verify-a-host-key) + to ensure it is correct. +1. To prevent force-pushing over diverged refs, select [**Keep divergent refs**](push.md#keep-divergent-refs). +1. Optional. Select [**Mirror only protected branches**](#mirror-only-protected-branches). +1. Select **Mirror repository**. + +If you select `SSH public key` as your authentication method, GitLab generates a +public key for your GitLab repository. You must provide this key to the non-GitLab server. +To learn more, read [Get your SSH public key](#get-your-ssh-public-key). + +## Update a mirror When the mirror repository is updated, all new branches, tags, and commits are visible in the -project's activity feed. +project's activity feed. A repository mirror at GitLab updates automatically. +You can also manually trigger an update: -Users with the [Maintainer role](../../../permissions.md) for the project can also force an -immediate update, unless: +- At most once every five minutes on GitLab.com. +- According to [the pull mirroring interval limit](../../../../administration/instance_limits.md#pull-mirroring-interval) + set by the administrator on self-managed instances. -- The mirror is already being updated. -- The [limit for pull mirroring interval seconds](../../../../administration/instance_limits.md#pull-mirroring-interval) has not elapsed after its last update. +### Force an update -For security reasons, the URL to the original repository is only displayed to users with the -[Maintainer role](../../../permissions.md) or the [Owner role](../../../permissions.md) for the mirrored -project. +While mirrors are scheduled to update automatically, you can force an immediate update unless: -## Use cases +- The mirror is already being updated. +- The [interval, in seconds](../../../../administration/instance_limits.md#pull-mirroring-interval) + for pull mirroring limits has not elapsed after its last update. -The following are some possible use cases for repository mirroring: +Prerequisite: -- You migrated to GitLab but still must keep your project in another source. In that case, you - can set it up to mirror to GitLab (pull) and all the essential history of commits, tags, - and branches are available in your GitLab instance. **(PREMIUM)** -- You have old projects in another source that you don't use actively anymore, but don't want to - remove for archiving purposes. In that case, you can create a push mirror so that your active - GitLab repository can push its changes to the old location. -- You are a GitLab self-managed user for privacy reasons and your instance is closed to the public, - but you still have certain software components that you want open sourced. In this case, utilizing - GitLab to be your primary repository which is closed from the public, and using push mirroring to a - GitLab.com repository that's public, allows you to open source specific projects and contribute back - to the open source community. +- You must have at least the [Maintainer role](../../../permissions.md) for the project. + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Mirroring repositories**. +1. Scroll to **Mirrored repositories** and identify the mirror to update. +1. Select **Update now** (**{retry}**): +  ## Mirror only protected branches **(PREMIUM)** > Moved to GitLab Premium in 13.9. -Based on the mirror direction that you choose, you can opt to mirror only the +You can choose to mirror only the [protected branches](../../protected_branches.md) in the mirroring project, -either from or to your remote repository. For pull mirroring, non-protected branches in -the mirroring project are not mirrored and can diverge. +either from or to your remote repository. For [pull mirroring](pull.md), +non-protected branches in the mirroring project are not mirrored and can diverge. + +To use this option, select **Only mirror protected branches** when you create a repository mirror. -To use this option, check the **Only mirror protected branches** box when -creating a repository mirror. **(PREMIUM)** +## Authentication methods for mirrors -## SSH authentication +When you create a mirror, you must configure the authentication method for it. +GitLab supports these authentication methods: -> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/-/merge_requests/22982) in GitLab 11.6 for Push mirroring. +- [SSH authentication](#ssh-authentication). +- Password. + +### SSH authentication SSH authentication is mutual: -- You have to prove to the server that you're allowed to access the repository. -- The server also has to prove to *you* that it's who it claims to be. +- You must prove to the server that you're allowed to access the repository. +- The server must also *prove to you* that it's who it claims to be. -You provide your credentials as a password or public key. The server that the -other repository resides on provides its credentials as a "host key", the -fingerprint of which needs to be verified manually. +For SSH authentication, you provide your credentials as a password or _public key_. +The server that the other repository resides on provides its credentials as a _host key_. +You must [verify the fingerprint](#verify-a-host-key) of this host key manually. If you're mirroring over SSH (using an `ssh://` URL), you can authenticate using: - Password-based authentication, just as over HTTPS. -- Public key authentication. This is often more secure than password authentication, +- Public key authentication. This method is often more secure than password authentication, especially when the other repository supports [deploy keys](../../deploy_keys/index.md). -To get started: - -1. In your project, go to **Settings > Repository**, and then expand the **Mirroring repositories** section. -1. Enter an `ssh://` URL for mirroring. +### Get your SSH public key -NOTE: -SCP-style URLs (that is, `git@example.com:group/project.git`) are not supported at this time. +When you mirror a repository and select the **SSH public key** as your +authentication method, GitLab generates a public key for you. The non-GitLab server +needs this key to establish trust with your GitLab repository. To copy your SSH public key: -Entering the URL adds two buttons to the page: +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Repository**. +1. Expand **Mirroring repositories**. +1. Scroll to **Mirrored repositories**. +1. Identify the correct repository, and select **Copy SSH public key** (**{copy-to-clipboard}**). +1. Add the public SSH key to the other repository's configuration: + - If the other repository is hosted on GitLab, add the public SSH key + as a [deploy key](../../../project/deploy_keys/index.md). + - If the other repository is hosted elsewhere, add the key to + your user's `authorized_keys` file. Paste the entire public SSH key into the + file on its own line and save it. -- **Detect host keys**. -- **Input host keys manually**. +If you must change the key at any time, you can remove and re-add the mirror +to generate a new key. Update the other repository with the new +key to keep the mirror running. -If you select the: +NOTE: +The generated keys are stored in the GitLab database, not in the file system. Therefore, +SSH public key authentication for mirrors cannot be used in a pre-receive hook. -- **Detect host keys** button, GitLab fetches the host keys from the server and display the fingerprints. -- **Input host keys manually** button, a field is displayed where you can paste in host keys. +### Verify a host key -Assuming you used the former, you now must verify that the fingerprints are -those you expect. GitLab.com and other code hosting sites publish their -fingerprints in the open for you to check: +When using a host key, always verify the fingerprints match what you expect. +GitLab.com and other code hosting sites publish their fingerprints +for you to check: - [AWS CodeCommit](https://docs.aws.amazon.com/codecommit/latest/userguide/regions.html#regions-fingerprints) - [Bitbucket](https://support.atlassian.com/bitbucket-cloud/docs/configure-ssh-and-two-step-verification/) @@ -112,9 +172,11 @@ fingerprints in the open for you to check: - [Savannah](http://savannah.gnu.org/maintenance/SshAccess/) - [SourceForge](https://sourceforge.net/p/forge/documentation/SSH%20Key%20Fingerprints/) -Other providers vary. If you're running self-managed GitLab, or otherwise -have access to the server for the other repository, you can securely gather the -key fingerprints: +Other providers vary. You can securely gather key fingerprints with the following +command if you: + +- Run self-managed GitLab. +- Have access to the server for the other repository. ```shell $ cat /etc/ssh/ssh_host*pub | ssh-keygen -E md5 -l -f - @@ -123,45 +185,9 @@ $ cat /etc/ssh/ssh_host*pub | ssh-keygen -E md5 -l -f - 2048 MD5:3f:72:be:3d:62:03:5c:62:83:e8:6e:14:34:3a:85:1d root@example.com (RSA) ``` -NOTE: -You must exclude `-E md5` for some older versions of SSH. - -When mirroring the repository, GitLab checks that at least one of the -stored host keys matches before connecting. This can prevent malicious code from -being injected into your mirror, or your password being stolen. - -### SSH public key authentication - -To use SSH public key authentication, you must also choose that option -from the **Authentication method** dropdown. When the mirror is created, -GitLab generates a 4096-bit RSA key that can be copied by selecting the **Copy SSH public key** button. - - +Older versions of SSH may require you to remove `-E md5` from the command. -You then must add the public SSH key to the other repository's configuration: - -- If the other repository is hosted on GitLab, you should add the public SSH key - as a [deploy key](../../../project/deploy_keys/index.md). -- If the other repository is hosted elsewhere, you must add the key to - your user's `authorized_keys` file. Paste the entire public SSH key into the - file on its own line and save it. - -If you must change the key at any time, you can remove and re-add the mirror -to generate a new key. Update the other repository with the new -key to keep the mirror running. - -NOTE: -The generated keys are stored in the GitLab database, not in the file system. Therefore, -SSH public key authentication for mirrors cannot be used in a pre-receive hook. - -## Force an update **(FREE)** - -While mirrors are scheduled to update automatically, you can always force an update by using the -update button which is available on the **Mirroring repositories** section of the **Repository Settings** page. - - - -## Resources +## Related topics - Configure a [Pull Mirroring Interval](../../../../administration/instance_limits.md#pull-mirroring-interval) - [Disable mirrors for a project](../../../admin_area/settings/visibility_and_access_controls.md#enable-project-mirroring) @@ -171,24 +197,33 @@ update button which is available on the **Mirroring repositories** section of th Should an error occur during a push, GitLab displays an **Error** highlight for that repository. Details on the error can then be seen by hovering over the highlight text. -### 13:Received RST_STREAM with error code 2 with GitHub +### Received RST_STREAM with error code 2 with GitHub + +If you receive this message while mirroring to a GitHub repository: + +```plaintext +13:Received RST_STREAM with error code 2 +``` + +Your GitHub settings might be set to block pushes that expose your email address +used in commits. To fix this problem, either: -If you receive a "13:Received RST_STREAM with error code 2" message while mirroring to a GitHub repository, -your GitHub settings might be set to block pushes that expose your email address used in commits. Either -set your email address on GitHub to be public, or disable the [Block command line pushes that expose my email](https://github.com/settings/emails) setting. +- Set your GitHub email address to public. +- Disable the [Block command line pushes that expose my email](https://github.com/settings/emails) setting. -### 4:Deadline Exceeded +### Deadline Exceeded -When upgrading to GitLab 11.11.8 or newer, a change in how usernames are represented means that you +When upgrading to GitLab 11.11.8 or later, a change in how usernames are represented means that you must update your mirroring username and password to ensure that `%40` characters are replaced with `@`. ### Connection blocked because server only allows public key authentication -As the error indicates, the connection is getting blocked between GitLab and the remote repository. Even if a -[TCP Check](../../../../administration/raketasks/maintenance.md#check-tcp-connectivity-to-a-remote-site) is successful, -you must check any networking components in the route from GitLab to the remote Server to ensure there's no blockage. +The connection between GitLab and the remote repository is blocked. Even if a +[TCP Check](../../../../administration/raketasks/maintenance.md#check-tcp-connectivity-to-a-remote-site) +is successful, you must check any networking components in the route from GitLab +to the remote server for blockage. -For example, we've seen this error when a Firewall was performing a `Deep SSH Inspection` on outgoing packets. +This error can occur when a firewall performs a `Deep SSH Inspection` on outgoing packets. ### Could not read username: terminal prompts disabled @@ -196,29 +231,31 @@ If you receive this error after creating a new project using [GitLab CI/CD for external repositories](../../../../ci/ci_cd_for_external_repos/): ```plaintext -"2:fetch remote: "fatal: could not read Username for 'https://bitbucket.org': terminal prompts disabled\n": exit status 128." +"2:fetch remote: "fatal: could not read Username for 'https://bitbucket.org': +terminal prompts disabled\n": exit status 128." ``` Check if the repository owner is specified in the URL of your mirrored repository: -1. Go to your project. +1. On the top bar, select **Menu > Projects** and find your project. 1. On the left sidebar, select **Settings > Repository**. -1. Select **Mirroring repositories**. -1. If no repository owner is specified, delete and add the URL again in this format: +1. Expand **Mirroring repositories**. +1. If no repository owner is specified, delete and add the URL again in this format, + replacing `OWNER`, `ACCOUNTNAME`, and `REPONAME` with your values: ```plaintext - https://**<repo_owner>**@bitbucket.org/<accountname>/<reponame>.git + https://OWNER@bitbucket.org/ACCOUNTNAME/REPONAME.git ``` -The repository owner is needed for Bitbucket to connect to the repository for mirroring. +When connecting to the repository for mirroring, Bitbucket requires the repository owner in the string. ### Pull mirror is missing LFS files In some cases, pull mirroring does not transfer LFS files. This issue occurs when: - You use an SSH repository URL. The workaround is to use an HTTPS repository URL instead. - There is [an issue to fix this for SSH URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/11997). + An issue exists [to fix this problem for SSH URLs](https://gitlab.com/gitlab-org/gitlab/-/issues/11997). - You're using GitLab 14.0 or older, and the source repository is a public Bitbucket URL. - This was [fixed in GitLab 14.0.6](https://gitlab.com/gitlab-org/gitlab/-/issues/335123). + [Fixed](https://gitlab.com/gitlab-org/gitlab/-/issues/335123) in GitLab 14.0.6. - You mirror an external repository using object storage. - There is [an issue to fix this](https://gitlab.com/gitlab-org/gitlab/-/issues/335495). + An issue exists [to fix this problem](https://gitlab.com/gitlab-org/gitlab/-/issues/335495). diff --git a/doc/user/project/repository/mirror/pull.md b/doc/user/project/repository/mirror/pull.md index d1943cbfd71..4c437d0f8c8 100644 --- a/doc/user/project/repository/mirror/pull.md +++ b/doc/user/project/repository/mirror/pull.md @@ -41,7 +41,7 @@ After you configure a GitLab repository as a pull mirror: 1. Sidekiq becomes available to process updates, mirrors are updated. If the update process: - **Succeeds**: An update is enqueued again with at least a 30 minute wait. - **Fails**: The update is attempted again later. After 14 failures, a mirror is marked as a - [hard failure](#hard-failure) and is no longer enqueued for updates. A branch diverging + [hard failure](#fix-hard-failures-when-mirroring) and is no longer enqueued for updates. A branch diverging from its upstream counterpart can cause failures. To prevent branches from diverging, configure [Overwrite diverged branches](#overwrite-diverged-branches) when you create your mirror. @@ -102,7 +102,7 @@ updates are pulled immediately. For more information, read [Start the pull mirroring process for a project](../../../../api/projects.md#start-the-pull-mirroring-process-for-a-project). -## Hard failure +## Fix hard failures when mirroring > Moved to GitLab Premium in 13.9. @@ -112,7 +112,7 @@ and mirroring attempts stop. This failure is visible in either the: - Project's main dashboard. - Pull mirror settings page. -You can resume the project mirroring again by [forcing an update](index.md#force-an-update). +To resume project mirroring, [force an update](index.md#force-an-update). ## Related topics diff --git a/doc/user/project/repository/vscode.md b/doc/user/project/repository/vscode.md new file mode 100644 index 00000000000..bbf14a71653 --- /dev/null +++ b/doc/user/project/repository/vscode.md @@ -0,0 +1,47 @@ +--- +stage: Create +group: Code Review +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# GitLab Workflow extension for VS Code **(FREE)** + +The [GitLab Workflow extension](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow) +integrates GitLab with Visual Studio Code. You can decrease context switching and +do more day-to-day tasks in Visual Studio Code, such as: + +- [View issues](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#browse-issues-review-mrs). +- Run [common commands](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#commands) + from the Visual Studio Code [command palette](https://code.visualstudio.com/docs/getstarted/userinterface#_command-palette). +- Create and [review](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#merge-request-reviews) + merge requests directly from Visual Studio Code. +- [Validate your GitLab CI configuration](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#validate-gitlab-ci-configuration). +- [View the status of your pipeline](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#information-about-your-branch-pipelines-mr-closing-issue). +- [Create](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#create-snippet) + and paste snippets to, and from, your editor. +- [Browse repositories](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#browse-a-repository-without-cloning) + without cloning them. + +## Download the extension + +Download the extension from the [Visual Studio Code Marketplace](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow). + +## Configure the extension + +After you [download the extension](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow) +you can [configure](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#extension-settings): + +- [Features to display or hide](https://gitlab.com/gitlab-org/gitlab-vscode-extension#extension-settings). +- [Self-signed certificate](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#self-signed-certificates) information. + +## Report issues with the extension + +Report any issues, bugs, or feature requests in the +[`gitlab-vscode-extension` issue queue](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/issues). + +## Related topics + +- [Download the extension](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow) +- [Extension documentation](https://gitlab.com/gitlab-org/gitlab-vscode-extension/-/blob/main/README.md) +- The extension source code is available in the + [`gitlab-vscode-extension`](https://gitlab.com/gitlab-org/gitlab-vscode-extension/) project. diff --git a/doc/user/project/repository/x509_signed_commits/index.md b/doc/user/project/repository/x509_signed_commits/index.md index 17031dd29af..2db7ae308bd 100644 --- a/doc/user/project/repository/x509_signed_commits/index.md +++ b/doc/user/project/repository/x509_signed_commits/index.md @@ -158,7 +158,7 @@ can start signing your tags: git config --global tag.gpgsign true ``` -## Resources +## Related topics - [Rake task for X.509 signatures](../../../../raketasks/x509_signatures.md) diff --git a/doc/user/project/requirements/index.md b/doc/user/project/requirements/index.md index 661bd3e0598..6ee23c91680 100644 --- a/doc/user/project/requirements/index.md +++ b/doc/user/project/requirements/index.md @@ -38,7 +38,9 @@ see [GitLab Requirements Traceability Walkthrough](https://youtu.be/VIiuTQYFVa0) A paginated list of requirements is available in each project, and there you can create a new requirement. -Users with Reporter or higher [permissions](../../permissions.md) can create requirements. +Prerequisite: + +- You must have at least the Reporter [role](../../permissions.md). To create a requirement: @@ -66,7 +68,9 @@ next to the requirement title. You can edit a requirement from the requirements list page. -Users with Reporter or higher [permissions](../../permissions.md) can edit requirements. +Prerequisite: + +- You must have at least the Reporter [role](../../permissions.md). To edit a requirement: @@ -80,7 +84,9 @@ To edit a requirement: You can archive an open requirement while you're in the **Open** tab. -Users with Reporter or higher [permissions](../../permissions.md) can archive requirements. +Prerequisite: + +- You must have at least the Reporter [role](../../permissions.md). To archive a requirement, select **Archive** (**{archive}**). @@ -90,7 +96,9 @@ As soon as a requirement is archived, it no longer appears in the **Open** tab. You can view the list of archived requirements in the **Archived** tab. -Users with Reporter or higher [permissions](../../permissions.md) can reopen archived requirements. +Prerequisite: + +- You must have at least the Reporter [role](../../permissions.md).  @@ -209,13 +217,13 @@ requirements_confirmation: > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/246857) in GitLab 13.7. +You must have at least the Reporter [role](../../permissions.md). + You can import requirements to a project by uploading a [CSV file](https://en.wikipedia.org/wiki/Comma-separated_values) with the columns `title` and `description`. After the import, the user uploading the CSV file is set as the author of the imported requirements. -Users with Reporter or higher [permissions](../../permissions.md) can import requirements. - ### Import the file Before you import your file: @@ -281,7 +289,9 @@ By exporting requirements, you and your team can import them into another tool o your customers. Exporting requirements can aid collaboration with higher-level systems, as well as audit and regulatory compliance tasks. -Users with Reporter or higher [permissions](../../permissions.md) can export requirements. +Prerequisite: + +- You must have at least the Reporter [role](../../permissions.md). To export requirements: diff --git a/doc/user/project/service_desk.md b/doc/user/project/service_desk.md index fa5ef35418a..4f8e068ca2d 100644 --- a/doc/user/project/service_desk.md +++ b/doc/user/project/service_desk.md @@ -86,10 +86,10 @@ To improve your project's security, we recommend the following: - [Enable Akismet](../../integration/akismet.md) on your GitLab instance to add spam checking to this service. Unblocked email spam can result in many spam issues being created. -The unique internal email address is visible to project members with Maintainer (or higher) -[permission level](../permissions.md) -in your GitLab instance. However, when using an email alias externally, an end user -(issue creator) cannot see the internal email address displayed in the information note. +The unique internal email address is visible to project members at least +the Reporter [role](../permissions.md) in your GitLab instance. +An external user (issue creator) cannot see the internal email address +displayed in the information note. ### Using customized email templates @@ -137,15 +137,23 @@ You can use these placeholders to be automatically replaced in each email: #### New Service Desk issues -You can select one [issue description template](description_templates.md#create-an-issue-template) +You can select one [description template](description_templates.md#create-an-issue-template) **per project** to be appended to every new Service Desk issue's description. -Issue description templates should reside in your repository's `.gitlab/issue_templates/` directory. -To use a custom issue template with Service Desk, in your project: +You can set description templates at various levels: -1. [Create a description template](description_templates.md#create-an-issue-template) -1. Go to **Settings > General > Service Desk**. -1. From the dropdown **Template to append to all Service Desk issues**, select your template. +- The entire [instance](description_templates.md#set-instance-level-description-templates). +- A specific [group or subgroup](description_templates.md#set-group-level-description-templates). +- A specific [project](description_templates.md#set-a-default-template-for-merge-requests-and-issues). + +The templates are inherited. For example, in a project, you can also access templates set for the instance or the project’s parent groups. + +To use a custom description template with Service Desk: + +1. On the top bar, select **Menu > Projects** and find your project. +1. [Create a description template](description_templates.md#create-an-issue-template). +1. On the left sidebar, select **Settings > General > Service Desk**. +1. From the dropdown **Template to append to all Service Desk issues**, search or select your template. ### Using a custom email display name @@ -156,7 +164,8 @@ this name in the `From` header. The default display name is `GitLab Support Bot` To edit the custom email display name: -1. In a project, go to **Settings > General > Service Desk**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > General > Service Desk**. 1. Enter a new name in **Email display name**. 1. Select **Save Changes**. @@ -166,13 +175,13 @@ To edit the custom email display name: > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/284656) in GitLab 13.8. It is possible to customize the email address used by Service Desk. To do this, you must configure -both a [custom mailbox](#configuring-a-custom-mailbox) and a +a [custom mailbox](#configuring-a-custom-mailbox). If you want you can also configure a [custom suffix](#configuring-a-custom-email-address-suffix). #### Configuring a custom mailbox NOTE: -On GitLab.com a custom mailbox is already configured with `contact-project+%{key}@incoming.gitlab.com` as the email address, so you only have to configure the +On GitLab.com a custom mailbox is already configured with `contact-project+%{key}@incoming.gitlab.com` as the email address, you can still configure the [custom suffix](#configuring-a-custom-email-address-suffix) in project settings. Using the `service_desk_email` configuration, you can customize the mailbox @@ -198,7 +207,7 @@ To configure a custom mailbox for Service Desk with IMAP, add the following snip service_desk_email: enabled: true address: "project_contact+%{key}@example.com" - user: "project_support@example.com" + user: "project_contact@example.com" password: "[REDACTED]" host: "imap.gmail.com" port: 993 @@ -215,7 +224,7 @@ To configure a custom mailbox for Service Desk with IMAP, add the following snip ```ruby gitlab_rails['service_desk_email_enabled'] = true gitlab_rails['service_desk_email_address'] = "project_contact+%{key}@gmail.com" - gitlab_rails['service_desk_email_email'] = "project_support@gmail.com" + gitlab_rails['service_desk_email_email'] = "project_contact@gmail.com" gitlab_rails['service_desk_email_password'] = "[REDACTED]" gitlab_rails['service_desk_email_mailbox_name'] = "inbox" gitlab_rails['service_desk_email_idle_timeout'] = 60 @@ -271,6 +280,8 @@ For example, suppose the `mygroup/myproject` project Service Desk settings has t The Service Desk email address for this project is: `contact+mygroup-myproject-support@example.com`. The [incoming email](../../administration/incoming_email.md) address still works. +If you don't configure the custom suffix, the default project identification will be used for identifying the project. You can see that email address in the project settings. + ## Using Service Desk You can use Service Desk to [create an issue](#as-an-end-user-issue-creator) or [respond to one](#as-a-responder-to-the-issue). diff --git a/doc/user/project/settings/img/general_settings_v13_11.png b/doc/user/project/settings/img/general_settings_v13_11.png Binary files differdeleted file mode 100644 index 9da5acdf82e..00000000000 --- a/doc/user/project/settings/img/general_settings_v13_11.png +++ /dev/null diff --git a/doc/user/project/settings/img/import_export_download_export.png b/doc/user/project/settings/img/import_export_download_export.png Binary files differindex c7ab7565fc7..62292e99e8e 100644 --- a/doc/user/project/settings/img/import_export_download_export.png +++ b/doc/user/project/settings/img/import_export_download_export.png diff --git a/doc/user/project/settings/img/import_export_export_button.png b/doc/user/project/settings/img/import_export_export_button.png Binary files differindex 6933e3edfcc..6f3663d64e8 100644 --- a/doc/user/project/settings/img/import_export_export_button.png +++ b/doc/user/project/settings/img/import_export_export_button.png diff --git a/doc/user/project/settings/import_export.md b/doc/user/project/settings/import_export.md index 6c3bf731cf8..74879dae2d6 100644 --- a/doc/user/project/settings/import_export.md +++ b/doc/user/project/settings/import_export.md @@ -2,7 +2,6 @@ stage: Manage group: Import info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" -type: reference, howto --- # Project import/export **(FREE)** @@ -159,7 +158,9 @@ To export a project and its data, follow these steps: 1. Go to your project's homepage. -1. Click **Settings** in the sidebar. +1. Select **Settings** in the sidebar. + +1. Scroll down and expand the **Advanced** section. 1. Scroll down to find the **Export project** button: @@ -178,12 +179,14 @@ To export a project and its data, follow these steps: ## Import the project +> Default maximum import file size [changed](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50 MB to unlimited in GitLab 13.8. + WARNING: Only import projects from sources you trust. If you import a project from an untrusted source, it may be possible for an attacker to steal your sensitive data. 1. The GitLab project import feature is the first import option when creating a - new project. Click on **GitLab export**: + new project. Select **GitLab export**:  @@ -191,7 +194,7 @@ may be possible for an attacker to steal your sensitive data.  -1. Click on **Import project** to begin importing. Your newly imported project +1. Select **Import project** to begin importing. Your newly imported project page appears shortly. NOTE: @@ -199,9 +202,8 @@ If use of the `Internal` visibility level [is restricted](../../../public_access/public_access.md#restrict-use-of-public-or-internal-projects), all imported projects are given the visibility of `Private`. -NOTE: -The maximum import file size can be set by the Administrator, default is `0` (unlimited). -As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the [Application settings API](../../../api/settings.md#change-application-settings) or the [Admin Area UI](../../admin_area/settings/account_and_limit_settings.md). Default [modified](https://gitlab.com/gitlab-org/gitlab/-/issues/251106) from 50MB to 0 in GitLab 13.8. +The maximum import file size can be set by the Administrator, and the default is `0` (unlimited). +As an administrator, you can modify the maximum import file size. To do so, use the `max_import_size` option in the [Application settings API](../../../api/settings.md#change-application-settings) or the [Admin Area UI](../../admin_area/settings/account_and_limit_settings.md). ### Project import status @@ -224,6 +226,11 @@ To help avoid abuse, by default, users are rate limited to: GitLab.com may have [different settings](../../gitlab_com/index.md#importexport) from the defaults. +## Automate group and project import **(PREMIUM)** + +For information on automating user, group, and project import API calls, see +[Automate group and project import](../import/index.md#automate-group-and-project-import). + ## Troubleshooting ### Project fails to import due to mismatch @@ -233,15 +240,18 @@ does not match between the exported project, and the project import, the project Review [issue 276930](https://gitlab.com/gitlab-org/gitlab/-/issues/276930), and either: - Ensure shared runners are enabled in both the source and destination projects. -- Disable shared runners on the parent group when you import the project. +- Disable shared runners on the parent group when you import the project. -### Import workaround for large repositories +### Import workarounds for large repositories [Maximum import size limitations](#import-the-project) -can prevent an import from being successful. -If changing the import limits is not possible, -the following local workflow can be used to temporarily -reduce the repository size for another import attempt. +can prevent an import from being successful. If changing the import limits is not possible, you can +try one of the workarounds listed here. + +#### Workaround option 1 + +The following local workflow can be used to temporarily +reduce the repository size for another import attempt: 1. Create a temporary working directory from the export: @@ -291,6 +301,58 @@ reduce the repository size for another import attempt. delete the temporary, `smaller-tmp-main` branch, and the local, temporary data. +#### Workaround option 2 + +Rather than attempting to push all changes at once, this workaround: + +- Separates the project import from the Git Repository import +- Incrementally pushes the repository to GitLab + +1. Make a local clone of the repository to migrate. In a later step, you push this clone outside of + the project export. +1. Download the export and remove the `project.bundle` (which contains the Git repository): + + ```shell + tar -czvf new_export.tar.gz --exclude='project.bundle' @old_export.tar.gz + ``` + +1. Import the export without a Git repository. It asks you to confirm to import without a + repository. +1. Save this bash script as a file and run it after adding the appropriate origin. + + ```shell + #!/bin/sh + + # ASSUMPTIONS: + # - The GitLab location is "origin" + # - The default branch is "main" + # - This will attempt to push in chunks of 500MB (dividing the total size by 500MB). + # Decrease this size to push in smaller chunks if you still receive timeouts. + + git gc + SIZE=$(git count-objects -v 2> /dev/null | grep size-pack | awk '{print $2}') + + # Be conservative... and try to push 2GB at a time + # (given this assumes each commit is the same size - which is wrong) + BATCHES=$(($SIZE / 500000)) + TOTAL_COMMITS=$(git rev-list --count HEAD) + if (( BATCHES > TOTAL_COMMITS )); then + BATCHES=$TOTAL_COMMITS + fi + + INCREMENTS=$(( ($TOTAL_COMMITS / $BATCHES) - 1 )) + + for (( BATCH=BATCHES; BATCH>=1; BATCH-- )) + do + COMMIT_NUM=$(( $BATCH - $INCREMENTS )) + COMMIT_SHA=$(git log -n $COMMIT_NUM --format=format:%H | tail -1) + git push -u origin ${COMMIT_SHA}:refs/heads/main + done + git push -u origin main + git push -u origin -—all + git push -u origin -—tags + ``` + ### Manually execute export steps Exports sometimes fail without giving enough information to troubleshoot. In these cases, it can be diff --git a/doc/user/project/settings/index.md b/doc/user/project/settings/index.md index 30def6ae80f..5c4d4649240 100644 --- a/doc/user/project/settings/index.md +++ b/doc/user/project/settings/index.md @@ -28,12 +28,32 @@ functionality of a project. Adjust your project's name, description, avatar, [default branch](../repository/branches/default.md), and topics: - - The project description also partially supports [standard Markdown](../../markdown.md#features-extended-from-standard-markdown). You can use [emphasis](../../markdown.md#emphasis), [links](../../markdown.md#links), and [line-breaks](../../markdown.md#line-breaks) to add more context to the project description. +#### Topics + +Use topics to categorize projects and find similar new projects. + +To assign topics to a project: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings** > **General**. +1. Under **Topics**, enter the project topics. Existing popular topics are suggested as you type. +1. Select **Save changes**. + +For GitLab.com, explore popular topics on the [Explore topics page](../working_with_projects.md#explore-topics). +When you select a topic, you can see relevant projects. + +NOTE: +The assigned topics are visible only to everyone with access to the project, +but everyone can see which topics exist at all on the GitLab instance. +Do not include sensitive information in the name of a topic. + +If you're an instance administrator, see also +[Administering topics](../../admin_area/index.md#administering-topics). + #### Compliance frameworks **(PREMIUM)** > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/276221) in GitLab 13.9. @@ -157,7 +177,9 @@ audit trail: include: # Execute individual project's configuration (if project contains .gitlab-ci.yml) project: '$CI_PROJECT_PATH' file: '$CI_CONFIG_PATH' - ref: '$CI_COMMIT_REF_NAME' # Must be defined or MR pipelines always use the use default branch. + ref: '$CI_COMMIT_REF_NAME' # Must be defined or MR pipelines always use the use default branch + rules: + - exists: '$CI_CONFIG_PATH' ``` ##### Ensure compliance jobs are always run @@ -236,7 +258,7 @@ Use the switches to enable or disable the following features: | **Wiki** | ✓ | Enables a separate system for [documentation](../wiki/). | | **Snippets** | ✓ | Enables [sharing of code and text](../../snippets.md). | | **Pages** | ✓ | Allows you to [publish static websites](../pages/). | -| **Operations** | ✓ | Control access to [operations dashboard](../../../operations/index.md). | +| **Operations** | ✓ | Control access to Operations-related features, including [Operations Dashboard](../../../operations/index.md), [Environments and Deployments](../../../ci/environments/index.md), [Feature Flags](../../../operations/feature_flags.md). | | **Metrics Dashboard** | ✓ | Control access to [metrics dashboard](../integrations/prometheus.md). | Some features depend on others: @@ -293,6 +315,7 @@ Set up your project's merge request settings: - Enable [require an associated issue from Jira](../../../integration/jira/issues.md#require-associated-jira-issue-for-merge-requests-to-be-merged). - Enable [`delete source branch after merge` option by default](../merge_requests/getting_started.md#deleting-the-source-branch). - Configure [suggested changes commit messages](../merge_requests/reviews/suggestions.md#configure-the-commit-message-for-applied-suggestions). +- Configure [merge commit message template](../merge_requests/commit_templates.md). - Configure [the default target project](../merge_requests/creating_merge_requests.md#set-the-default-target-project) for merge requests coming from forks. ### Service Desk diff --git a/doc/user/project/settings/project_access_tokens.md b/doc/user/project/settings/project_access_tokens.md index cae9276eafd..85e412e7a41 100644 --- a/doc/user/project/settings/project_access_tokens.md +++ b/doc/user/project/settings/project_access_tokens.md @@ -10,6 +10,7 @@ type: reference, howto > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/210181) in GitLab 13.0. > - [Became available on GitLab.com](https://gitlab.com/gitlab-org/gitlab/-/issues/235765) in GitLab 13.5 for paid groups only. > - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/issues/235765) in GitLab 13.5. +> - [Changed](https://gitlab.com/gitlab-org/gitlab/-/issues/342327) in GitLab 14.5. Default prefix added. Project access tokens are similar to [personal access tokens](../../profile/personal_access_tokens.md) except they are attached to a project rather than a user. They can be used to: @@ -32,6 +33,9 @@ Project access tokens: For examples of how you can use a project access token to authenticate with the API, see the [relevant section from our API Docs](../../../api/index.md#personalproject-access-tokens). +NOTE: +For GitLab.com and self-managed instances, the default prefix is `glpat-`. + ## Creating a project access token 1. Log in to GitLab. @@ -58,7 +62,9 @@ For the bot: - The name is set to the name of the token. - The username is set to `project_{project_id}_bot` for the first access token, such as `project_123_bot`. -- The username is set to `project_{project_id}_bot{bot_count}` for further access tokens, such as `project_123_bot1`. +- The email is set to `project{project_id}_bot@example.com`, for example `project123_bot@example.com`. +- For additional access tokens in the same project, the username is set to `project_{project_id}_bot{bot_count}`, for example `project_123_bot1`. +- For additional acess tokens in the same project, the email is set to `project{project_id}_bot{bot_count}@example.com`, for example `project123_bot1@example.com` API calls made with a project access token are associated with the corresponding bot user. diff --git a/doc/user/project/web_ide/index.md b/doc/user/project/web_ide/index.md index 71cf0c03549..d75a180492e 100644 --- a/doc/user/project/web_ide/index.md +++ b/doc/user/project/web_ide/index.md @@ -15,7 +15,8 @@ projects by providing an advanced editor with commit staging. ## Open the Web IDE -You can open the Web IDE when viewing a file, from the repository file list, +Use the <kbd>.</kbd> [keyboard shortcut](../../shortcuts.md) to open the Web IDE. +You can also open the Web IDE when viewing a file, from the repository file list, and from merge requests: - *When viewing a file, or the repository file list* - @@ -100,12 +101,13 @@ same core features for highlighting and linking to particular lines in the edite ## Schema based validation -> - Support for validation based on predefined schemas [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218472) in GitLab 13.2. -> - It was deployed behind a feature flag, disabled by default. -> - It's enabled on GitLab.com. -> - It cannot be enabled or disabled per-project. -> - For GitLab self-managed instances, GitLab administrators can opt to [enable it](#enable-or-disable-validation-based-on-predefined-schemas). -> - Support for validation based on custom schemas [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/226982) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/218472) validation based on predefined schemas in GitLab 13.2 [with a flag](../../../administration/feature_flags.md) named `schema_linting`. Disabled by default. + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available for your entire instance, ask an administrator to [enable the feature flag](../../../administration/feature_flags.md) named `schema_linting`. +This feature cannot be enabled or disabled per-project. +On GitLab.com, this feature is available. +You should not use this feature for production environments. The Web IDE provides validation support for certain JSON and YAML files using schemas based on the [JSON Schema Store](https://www.schemastore.org/json/). @@ -115,28 +117,9 @@ based on the [JSON Schema Store](https://www.schemastore.org/json/). The Web IDE has validation for certain files built in. This feature is only supported for the `*.gitlab-ci.yml` files. -#### Enable or disable validation based on predefined schemas **(FREE SELF)** - -Validation based on predefined schemas is under development and not ready for production use. It is -deployed behind a feature flag that is **disabled by default** for self-managed instances, -[GitLab administrators with access to the GitLab Rails console](../../../administration/feature_flags.md) -can enable it for your instance. - -To enable it: - -```ruby -Feature.enable(:schema_linting) -``` - -To disable it: - -```ruby -Feature.disable(:schema_linting) -``` - ### Custom schemas **(PREMIUM)** -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/226982) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.4. +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/226982) in GitLab 13.4. The Web IDE also allows you to define custom schemas for certain JSON/YAML files in your project. You can do so by defining a `schemas` entry in the `.gitlab/.gitlab-webide.yml` file inside the diff --git a/doc/user/project/wiki/group.md b/doc/user/project/wiki/group.md new file mode 100644 index 00000000000..731fed2595a --- /dev/null +++ b/doc/user/project/wiki/group.md @@ -0,0 +1,71 @@ +--- +stage: Create +group: Editor +info: "To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments" +type: reference, how-to +--- + +# Group wikis **(PREMIUM)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13195) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.5. + +If you use GitLab groups to manage multiple projects, some of your documentation +might span multiple groups. You can create group wikis, instead of [project wikis](index.md), +to ensure all group members have the correct access permissions to contribute. +Group wikis are similar to [project wikis](index.md), with a few limitations: + +- [Git LFS](../../../topics/git/lfs/index.md) is not supported. +- Group wikis are not included in [global search](../../search/advanced_search.md). +- Changes to group wikis don't show up in the [group's activity feed](../../group/index.md#group-activity-analytics). +- Group wikis are enabled by default for **(PREMIUM)** and higher tiers. + You [can't turn them off from the GitLab user interface](https://gitlab.com/gitlab-org/gitlab/-/issues/208413). + +For updates, follow [the epic that tracks feature parity with project wikis](https://gitlab.com/groups/gitlab-org/-/epics/2782). + +Similar to project wikis, group members with the [Developer role](../../permissions.md#group-members-permissions) +and higher can edit group wikis. Group wiki repositories can be moved using the +[Group repository storage moves API](../../../api/group_repository_storage_moves.md). + +## View a group wiki + +To access a group wiki: + +1. On the top bar, select **Menu > Groups** and find your group. +1. To display the wiki, either: + - On the left sidebar, select **Wiki**. + - On any page in the project, use the <kbd>g</kbd> + <kbd>w</kbd> + [wiki keyboard shortcut](../../shortcuts.md). + +## Export a group wiki + +> Introduced in [GitLab 13.9](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/53247). + +Users with the [Owner role](../../permissions.md) in a group can +[import and export group wikis](../../group/settings/import_export.md) when importing +or exporting a group. + +Content created in a group wiki is not deleted when an account is downgraded or a +GitLab trial ends. The group wiki data is exported whenever the group owner of +the wiki is exported. + +To access the group wiki data from the export file if the feature is no longer +available, you have to: + +1. Extract the [export file tarball](../../group/settings/import_export.md) with + this command, replacing `FILENAME` with your file's name: + `tar -xvzf FILENAME.tar.gz` +1. Browse to the `repositories` directory. This directory contains a + [Git bundle](https://git-scm.com/docs/git-bundle) with the extension `.wiki.bundle`. +1. Clone the Git bundle into a new repository, replacing `FILENAME` with + your bundle's name: `git clone FILENAME.wiki.bundle` + +All files in the wiki are available in this Git repository. + +## Related topics + +- [Wiki settings for administrators](../../../administration/wikis/index.md) +- [Project wikis API](../../../api/wikis.md) +- [Group repository storage moves API](../../../api/group_repository_storage_moves.md) +- [Group wikis API](../../../api/group_wikis.md) +- [Wiki keyboard shortcuts](../../shortcuts.md#wiki-pages) +- [Epic: Feature parity with project wikis](https://gitlab.com/groups/gitlab-org/-/epics/2782) diff --git a/doc/user/project/wiki/index.md b/doc/user/project/wiki/index.md index e2a8167b14c..5d2a0530f68 100644 --- a/doc/user/project/wiki/index.md +++ b/doc/user/project/wiki/index.md @@ -12,16 +12,6 @@ to keep it in the same project as your code, you can use the wiki GitLab provide in each GitLab project. Every wiki is a separate Git repository, so you can create wiki pages in the web interface, or [locally using Git](#create-or-edit-wiki-pages-locally). -To access the wiki for a project or group, go to the page for your project or group -and either: - -- In the left sidebar, select **Wiki**. -- On any page in the project, use the <kbd>g</kbd> + <kbd>w</kbd> - [wiki keyboard shortcut](../../shortcuts.md). - -If **Wiki** is not listed in the left sidebar of your project, a project administrator -has [disabled it](#enable-or-disable-a-project-wiki). - GitLab wikis support Markdown, RDoc, AsciiDoc, and Org for content. Wiki pages written in Markdown support all [Markdown features](../../markdown.md), and also provide some [wiki-specific behavior](../../markdown.md#wiki-specific-markdown) @@ -35,6 +25,19 @@ with sibling pages listed in alphabetical order. To view a list of all pages, se  +## View a project wiki + +To access a project wiki: + +1. On the top bar, select **Menu > Projects** and find your project. +1. To display the wiki, either: + - On the left sidebar, select **Wiki**. + - On any page in the project, use the <kbd>g</kbd> + <kbd>w</kbd> + [wiki keyboard shortcut](../../shortcuts.md). + +If **Wiki** is not listed in the left sidebar of your project, a project administrator +has [disabled it](#enable-or-disable-a-project-wiki). + ## Configure a default branch for your wiki > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/221159) in GitLab 14.1. @@ -56,7 +59,10 @@ When a wiki is created, it is empty. On your first visit, you can create the home page users see when viewing the wiki. This page requires a specific title to be used as your wiki's home page. To create it: -1. Go to your project or group and select **Wiki**. +1. On the top bar, select **Menu**. + - For project wikis, select **Projects** and find your project. + - For group wikis, select **Groups** and find your group. +1. On the left sidebar, select **Wiki**. 1. Select **Create your first page**. 1. GitLab requires this first page be titled `home`. The page with this title serves as the front page for your wiki. @@ -71,7 +77,10 @@ to be used as your wiki's home page. To create it: Users with the [Developer role](../../permissions.md) can create new wiki pages: -1. Go to your project or group and select **Wiki**. +1. On the top bar, select **Menu**. + - For project wikis, select **Projects** and find your project. + - For group wikis, select **Groups** and find your group. +1. On the left sidebar, select **Wiki**. 1. Select **New page** on this page, or any other wiki page. 1. Select a content format. 1. Add a title for your new page. Page titles use @@ -135,7 +144,10 @@ may not be able to check out the wiki locally afterward. You need at least the [Developer role](../../permissions.md) to edit a wiki page: -1. Go to your project or group and select **Wiki**. +1. On the top bar, select **Menu**. + - For project wikis, select **Projects** and find your project. + - For group wikis, select **Groups** and find your group. +1. On the left sidebar, select **Wiki**. 1. Go to the page you want to edit, and either: - Use the <kbd>e</kbd> wiki [keyboard shortcut](../../shortcuts.md#wiki-pages). - Select the edit icon (**{pencil}**). @@ -151,7 +163,10 @@ For an example, read [Table of contents](../../markdown.md#table-of-contents). You need at least the [Developer role](../../permissions.md) to delete a wiki page: -1. Go to your project or group and select **Wiki**. +1. On the top bar, select **Menu**. + - For project wikis, select **Projects** and find your project. + - For group wikis, select **Groups** and find your group. +1. On the left sidebar, select **Wiki**. 1. Go to the page you want to delete. 1. Select the edit icon (**{pencil}**). 1. Select **Delete page**. @@ -161,7 +176,10 @@ You need at least the [Developer role](../../permissions.md) to delete a wiki pa You need at least the [Developer role](../../permissions.md) to move a wiki page: -1. Go to your project or group and select **Wiki**. +1. On the top bar, select **Menu**. + - For project wikis, select **Projects** and find your project. + - For group wikis, select **Groups** and find your group. +1. On the left sidebar, select **Wiki**. 1. Go to the page you want to move. 1. Select the edit icon (**{pencil}**). 1. Add the new path to the **Title** field. For example, if you have a wiki page @@ -172,9 +190,7 @@ You need at least the [Developer role](../../permissions.md) to move a wiki page ## View history of a wiki page The changes of a wiki page over time are recorded in the wiki's Git repository. -To view the changes for a wiki page, select **Page history**. - -From the history page you can see: +The history page shows:  @@ -184,13 +200,25 @@ From the history page you can see: - The last update. - Previous revisions, by selecting a revision number in the **Page version** column. +To view the changes for a wiki page: + +1. On the top bar, select **Menu**. + - For project wikis, select **Projects** and find your project. + - For group wikis, select **Groups** and find your group. +1. On the left sidebar, select **Wiki**. +1. Go to the page you want to view history for. +1. Select **Page history**. + ### View changes between page versions > [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/15242) in GitLab 13.2. You can see the changes made in a version of a wiki page, similar to versioned diff file views: -1. Go to your project or group and select **Wiki**. +1. On the top bar, select **Menu**. + - For project wikis, select **Projects** and find your project. + - For group wikis, select **Groups** and find your group. +1. On the left sidebar, select **Wiki**. 1. Go to the wiki page you're interested in. 1. Select **Page history** to see all page versions. 1. Select the commit message in the **Changes** column for the version you're interested in. @@ -203,10 +231,12 @@ You can see the changes made in a version of a wiki page, similar to versioned d > - Git events were [introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/216014) in **GitLab 13.0.** > - [Feature flag for Git events was removed](https://gitlab.com/gitlab-org/gitlab/-/issues/258665) in **GitLab 13.5** -GitLab tracks wiki creation, deletion, and update events. These events are displayed on the -[user profile](../../profile/index.md#access-your-user-profile), -[group](../../group/index.md#view-group-activity), -and [project](../working_with_projects.md#project-activity) activity pages. +GitLab tracks wiki creation, deletion, and update events. These events are displayed on these pages: + +- [User profile](../../profile/index.md#access-your-user-profile). +- Activity pages, depending on the type of wiki: + - [Group activity](../../group/index.md#view-group-activity). + - [Project activity](../working_with_projects.md#project-activity). Commits to wikis are not counted in [repository analytics](../../analytics/repository_analytics.md). @@ -218,7 +248,10 @@ You need at least the [Developer role](../../permissions.md) to customize the wi navigation sidebar. This process creates a wiki page named `_sidebar` which fully replaces the default sidebar navigation: -1. Go to your project or group and select **Wiki**. +1. On the top bar, select **Menu**. + - For project wikis, select **Projects** and find your project. + - For group wikis, select **Groups** and find your group. +1. On the left sidebar, select **Wiki**. 1. In the top right corner of the page, select **Edit sidebar**. 1. When complete, select **Save changes**. @@ -241,42 +274,20 @@ Support for displaying a generated table of contents with a custom side navigati ## Enable or disable a project wiki Wikis are enabled by default in GitLab. Project [administrators](../../permissions.md) -can enable or disable the project wiki by following the instructions in +can enable or disable a project wiki by following the instructions in [Sharing and permissions](../settings/index.md#sharing-and-permissions). Administrators for self-managed GitLab installs can [configure additional wiki settings](../../../administration/wikis/index.md). -## Group wikis **(PREMIUM)** - -> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/13195) in [GitLab Premium](https://about.gitlab.com/pricing/) 13.5. - -Group wikis work the same way as project wikis. Their usage is similar to project -wikis, with a few limitations: - -- Git LFS is not supported. -- Group wikis are not included in global search. -- Changes to group wikis don't show up in the group's activity feed. - -For updates, follow [the epic that tracks feature parity with project wikis](https://gitlab.com/groups/gitlab-org/-/epics/2782). - -Group wikis can be edited by members with the [Developer role](../../permissions.md#group-members-permissions) -and above. Group wiki repositories can be moved using the -[Group repository storage moves API](../../../api/group_repository_storage_moves.md). - -### Export a group wiki **(PREMIUM)** - -Users with the [Owner role](../../permissions.md) in a group can -[import and export group wikis](../../group/settings/import_export.md) when importing -or exporting a group. - -Content created in a group wiki is not deleted when an account is downgraded or a GitLab trial ends. +You can't disable [group wikis](group.md) from the GitLab user interface. ## Link an external wiki To add a link to an external wiki from a project's left sidebar: -1. Go to your project and select **Settings > Integrations**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **External wiki**. 1. Add the URL to your external wiki. 1. (Optional) Select **Test settings** to verify the connection. @@ -291,7 +302,8 @@ To hide the internal wiki from the sidebar, [disable the project's wiki](#disabl To hide the link to an external wiki: -1. Go to your project and select **Settings > Integrations**. +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Settings > Integrations**. 1. Select **External wiki**. 1. In the **Enable integration** section, clear the **Active** checkbox. 1. Select **Save changes**. @@ -300,6 +312,7 @@ To hide the link to an external wiki: To disable a project's internal wiki: +1. On the top bar, select **Menu > Projects** and find your project. 1. Go to your project and select **Settings > General**. 1. Expand **Visibility, project features, permissions**. 1. Scroll down to find **Wiki** and toggle it off (in gray). @@ -356,7 +369,7 @@ For the status of the ongoing development for CommonMark and GitLab Flavored Mar - [Basic Markdown formatting extensions](https://gitlab.com/groups/gitlab-org/-/epics/5404) epic. - [GitLab Flavored Markdown extensions](https://gitlab.com/groups/gitlab-org/-/epics/5438) epic. -## Resources +## Related topics - [Wiki settings for administrators](../../../administration/wikis/index.md) - [Project wikis API](../../../api/wikis.md) diff --git a/doc/user/project/working_with_projects.md b/doc/user/project/working_with_projects.md index 11b570f19e5..3c82e544e26 100644 --- a/doc/user/project/working_with_projects.md +++ b/doc/user/project/working_with_projects.md @@ -13,7 +13,7 @@ code are saved in projects, and most features are in the scope of projects. You can explore other popular projects available on GitLab. To explore projects: -1. On the top bar, select **Menu > Project**. +1. On the top bar, select **Menu > Projects**. 1. Select **Explore projects**. GitLab displays a list of projects, sorted by last updated date. To view @@ -25,12 +25,27 @@ By default, `/explore` is visible to unauthenticated users. However, if the [**Public** visibility level](../admin_area/settings/visibility_and_access_controls.md#restrict-visibility-levels) is restricted, `/explore` is visible only to signed-in users. +## Explore topics + +You can explore popular project topics available on GitLab. To explore project topics: + +1. On the top bar, select **Menu > Projects**. +1. Select **Explore topics**. + +GitLab displays a list of topics sorted by the number of associated projects. +To view projects associated with a topic, select a topic from the list. + +You can assign topics to a project on the [Project Settings page](settings/index.md#topics). + +If you're an instance administrator, you can administer all project topics from the +[Admin Area's Topics page](../admin_area/index.md#administering-topics). + ## Create a project To create a project in GitLab: -1. In your dashboard, click the green **New project** button or use the plus - icon in the navigation bar. This opens the **New project** page. +1. In your dashboard, select **New project** or use the **New...** **{plus-square}** icon +on the top bar. The **New project** page opens. 1. On the **New project** page, choose if you want to: - Create a [blank project](#blank-projects). - Create a project using one of the available [project templates](#project-templates). @@ -213,7 +228,7 @@ To star a project: To view your starred projects: -1. On the top bar, select **Menu > Project**. +1. On the top bar, select **Menu > Projects**. 1. Select **Starred Projects**. 1. GitLab displays information about your starred projects, including: diff --git a/doc/user/report_abuse.md b/doc/user/report_abuse.md index a8107d77113..810e1427e11 100644 --- a/doc/user/report_abuse.md +++ b/doc/user/report_abuse.md @@ -19,6 +19,7 @@ You can report a user through their: - [Profile](#reporting-abuse-through-a-users-profile) - [Comments](#reporting-abuse-through-a-users-comment) - [Issues and Merge requests](#reporting-abuse-through-a-users-issue-or-merge-request) +- [Snippets](snippets.md#mark-snippet-as-spam) ## Reporting abuse through a user's profile diff --git a/doc/user/search/advanced_search.md b/doc/user/search/advanced_search.md index f994539b9fc..f68951badff 100644 --- a/doc/user/search/advanced_search.md +++ b/doc/user/search/advanced_search.md @@ -26,7 +26,7 @@ when searching in: - Comments - Code - Commits -- Wiki +- Wiki (except [group wikis](../project/wiki/group.md)) - Users The Advanced Search can be useful in various scenarios: @@ -139,4 +139,4 @@ its performance: | Commits | `global_search_commits_tab` | When enabled, the global search includes commits as part of the search. | | Issues | `global_search_issues_tab` | When enabled, the global search includes issues as part of the search. | | Merge Requests | `global_search_merge_requests_tab` | When enabled, the global search includes merge requests as part of the search. | -| Wiki | `global_search_wiki_tab` | When enabled, the global search includes wiki as part of the search. | +| Wiki | `global_search_wiki_tab` | When enabled, the global search includes wiki as part of the search. [Group wikis](../project/wiki/group.md) are not included. | diff --git a/doc/user/search/img/basic_search.png b/doc/user/search/img/basic_search_v14_4.png Binary files differindex 86aab68f1f0..86aab68f1f0 100644 --- a/doc/user/search/img/basic_search.png +++ b/doc/user/search/img/basic_search_v14_4.png diff --git a/doc/user/search/img/issues_mrs_shortcut.png b/doc/user/search/img/issues_mrs_shortcut.png Binary files differdeleted file mode 100644 index 5be8079030a..00000000000 --- a/doc/user/search/img/issues_mrs_shortcut.png +++ /dev/null diff --git a/doc/user/search/img/issues_mrs_shortcut_v14_4.png b/doc/user/search/img/issues_mrs_shortcut_v14_4.png Binary files differnew file mode 100644 index 00000000000..2610e611446 --- /dev/null +++ b/doc/user/search/img/issues_mrs_shortcut_v14_4.png diff --git a/doc/user/search/index.md b/doc/user/search/index.md index 7cf62f09303..325e5386417 100644 --- a/doc/user/search/index.md +++ b/doc/user/search/index.md @@ -35,7 +35,7 @@ You can also filter the results using the search and filter field, as described GitLab shows shortcuts to issues and merge requests created by you or assigned to you in the search field in the upper right corner: - + ### Filter issue and merge request lists @@ -118,7 +118,7 @@ You can add this URL to your feed reader. You can filter the **Issues** list to individual instances by their ID. For example, enter filter `#10` to return only issue 10. The same applies to the **Merge Requests** list. Enter filter `#30` to return only merge request 30. - + ### Filtering merge requests by approvers **(PREMIUM)** @@ -290,7 +290,7 @@ To start a search, type into the search bar on the top-right of the screen. You in all GitLab and may also see the options to search in a group or project if you are in the group or project dashboard. - + After the results are returned, you can modify the search, select a different type of data to search, or choose a specific group or project. @@ -321,9 +321,10 @@ GitLab instance. ## Search settings -> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292941) in GitLab 13.8 behind a feature flag, disabled by default. +> - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/292941) in GitLab 13.8 [with a flag](../../administration/feature_flags.md) named `search_settings_in_page`. Disabled by default. > - [Added to Group, Administrator, and User settings](https://gitlab.com/groups/gitlab-org/-/epics/4842) in GitLab 13.9. -> - [Enabled by default](https://gitlab.com/gitlab-org/gitlab/-/issues/294025) in GitLab 13.11. +> - [Feature flag `search_settings_in_page` removed](https://gitlab.com/gitlab-org/gitlab/-/issues/294025) in GitLab 13.11. +> - [Generally available](https://gitlab.com/gitlab-org/gitlab/-/issues/294025) in GitLab 13.11. You can search inside a Project, Group, Administrator, or User's settings by entering a search term in the search box located at the top of the page. The search results diff --git a/doc/user/shortcuts.md b/doc/user/shortcuts.md index f46c5428248..d6cbbf352fc 100644 --- a/doc/user/shortcuts.md +++ b/doc/user/shortcuts.md @@ -43,6 +43,7 @@ These shortcuts are available in most areas of GitLab: | <kbd>Shift</kbd> + <kbd>t</kbd> | Go to your To-Do List page. | | <kbd>p</kbd> + <kbd>b</kbd> | Show or hide the Performance Bar. | | <kbd>g</kbd> + <kbd>x</kbd> | Toggle between [GitLab](https://gitlab.com/) and [GitLab Next](https://next.gitlab.com/) (GitLab SaaS only). | +| <kbd>.</kbd> | Open the [Web IDE](project/web_ide/index.md). | Additionally, the following shortcuts are available when editing text in text fields (for example, comments, replies, issue descriptions, and merge request @@ -101,6 +102,7 @@ These shortcuts are available when viewing issues and [merge requests](project/m | <kbd>]</kbd> or <kbd>j</kbd> | Move to next file (merge requests only). | | <kbd>[</kbd> or <kbd>k</kbd> | Move to previous file (merge requests only). | | <kbd>b</kbd> | Copy source branch name (merge requests only). | +| <kbd>.</kbd> | Open the [Web IDE](project/web_ide/index.md). | ### Project files @@ -114,6 +116,7 @@ These shortcuts are available when browsing the files in a project (go to | <kbd>enter</kbd> | Open selection. | | <kbd>Escape</kbd> | Go back to file list screen (only while searching for files, **Repository > Files**, then select **Find File**). | | <kbd>y</kbd> | Go to file permalink (only while viewing a file). | +| <kbd>.</kbd> | Open the [Web IDE](project/web_ide/index.md). | ### Web IDE diff --git a/doc/user/snippets.md b/doc/user/snippets.md index 64da024f5ba..e2cb9937f76 100644 --- a/doc/user/snippets.md +++ b/doc/user/snippets.md @@ -14,6 +14,9 @@ You can [comment on](#comment-on-snippets), [clone](#clone-snippets), and [syntax highlighting](#filenames), [embedding](#embed-snippets), [downloading](#download-snippets), and you can maintain your snippets with the [snippets API](../api/snippets.md). +You can create and manage your snippets through the GitLab user interface, or by +using the [GitLab Workflow VS Code extension](project/repository/vscode.md). +  GitLab provides two types of snippets: @@ -39,6 +42,8 @@ You can create snippets in multiple ways, depending on whether you want to creat - *For all other pages,* select the plus icon (**{plus-square-o}**) in the top navigation bar, then select **New snippet** from the dropdown menu. + - If you installed the [GitLab Workflow VS Code extension](project/repository/vscode.md), + use the [`Gitlab: Create snippet` command](https://marketplace.visualstudio.com/items?itemName=GitLab.gitlab-workflow#create-snippet). - **To create a project snippet**: Go to your project's page. Select the plus icon (**{plus-square-o}**), and then select **New snippet** from the **This project** section of the dropdown menu. @@ -127,7 +132,7 @@ A single snippet can support up to 10 files, which helps keep related files toge If you need more than 10 files for your snippet, we recommend you create a [wiki](project/wiki/index.md) instead. Wikis are available for projects at all -subscription levels, and [groups](project/wiki/index.md#group-wikis) for +subscription levels, and [groups](project/wiki/group.md) for [GitLab Premium](https://about.gitlab.com/pricing/). Snippets with multiple files display a file count in the [snippet list](https://gitlab.com/dashboard/snippets): @@ -206,6 +211,24 @@ snippet was created using the GitLab web interface the original line ending is W With snippets, you engage in a conversation about that piece of code, which can encourage user collaboration. +## Mark snippet as spam **(FREE SELF)** + +Administrators on self-managed GitLab instances can mark snippets as spam. + +Prerequisites: + +- You must be the administrator for your instance. +- [Akismet](../integration/akismet.md) spam protection must be enabled on the instance. + +To do this task: + +1. On the top bar, select **Menu > Projects** and find your project. +1. On the left sidebar, select **Snippets**. +1. Select the snippet you want to report as spam. +1. Select **Submit as spam**. + +GitLab forwards the spam to Akismet. + ## Troubleshooting ### Snippet limitations diff --git a/doc/user/tasks.md b/doc/user/tasks.md new file mode 100644 index 00000000000..e5e5510407e --- /dev/null +++ b/doc/user/tasks.md @@ -0,0 +1,34 @@ +--- +stage: Plan +group: Project Management +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/engineering/ux/technical-writing/#assignments +--- + +# Tasks **(FREE)** + +> [Introduced](https://gitlab.com/gitlab-org/gitlab/-/issues/334812) in GitLab 14.5 [with a flag](../administration/feature_flags.md) named `work_items`. Disabled by default. + +WARNING: +Tasks are in **Alpha**. Current implementation does not have any API requests and works with mocked data. +For the latest updates, check the [Tasks Roadmap](https://gitlab.com/groups/gitlab-org/-/epics/7103). + +FLAG: +On self-managed GitLab, by default this feature is not available. To make it available, +ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `work_items`. +The feature is not ready for production use. + +Use tasks to track steps needed for the [issue](project/issues/index.md) to be closed. + +When planning an issue, you need a way to capture and break down technical +requirements or steps necessary to complete it. An issue with tasks is better defined, +and so you can provide a more accurate issue weight and completion criteria. + +To see the roadmap for migrating issues and [epics](group/epics/index.md) +to work items and adding custom work item types, visit +[epic 6033](https://gitlab.com/groups/gitlab-org/-/epics/6033) or +[Plan direction page](https://about.gitlab.com/direction/plan/). + +## View a task + +The only way to view a task is to open it with a deep link, +for example: `/<group_name>/<project_name>/-/work_item/1` diff --git a/doc/user/usage_quotas.md b/doc/user/usage_quotas.md index 5a48353c9d4..16b6424b232 100644 --- a/doc/user/usage_quotas.md +++ b/doc/user/usage_quotas.md @@ -36,9 +36,8 @@ namespace to recalculate the storage. > - [Introduced](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/68898) project-level graph in GitLab 14.4 [with a flag](../administration/feature_flags.md) named `project_storage_ui`. Disabled by default. > - Enabled on GitLab.com in GitLab 14.4. - -FLAG: -On self-managed GitLab, by default this feature is not available. To make it available, ask an administrator to [enable the feature flag](../administration/feature_flags.md) named `project_storage_ui`. On GitLab.com, this feature is available. +> - Enabled on self-managed in GitLab 14.5. +> - [Feature flag removed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/71270) in GitLab 14.5. The following storage usage statistics are available to an owner: diff --git a/doc/user/workspace/index.md b/doc/user/workspace/index.md index d75d91f087d..8ca7f7defb2 100644 --- a/doc/user/workspace/index.md +++ b/doc/user/workspace/index.md @@ -46,3 +46,7 @@ The following provide a preview to the Workspace concept.   + +## Related topics + +- [Workspaces developer documentation](../../development/workspaces/index.md) |
